Oct 30 19:39:02.215634 ip-10-0-130-41 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:39:02.215645 ip-10-0-130-41 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:39:02.215652 ip-10-0-130-41 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:39:02.215896 ip-10-0-130-41 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:39:12.299819 ip-10-0-130-41 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:39:12.299833 ip-10-0-130-41 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 95106406f6a346a09648d8e64be07aac -- Oct 30 19:41:03.337953 ip-10-0-130-41 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:41:03.749785 ip-10-0-130-41 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:03.749785 ip-10-0-130-41 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:41:03.749785 ip-10-0-130-41 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:03.749785 ip-10-0-130-41 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:03.753265 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753191 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.753265 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753264 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.753265 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753269 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753273 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753276 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753279 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753281 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753292 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753295 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753298 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753301 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753303 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753306 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753309 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753311 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753314 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753316 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753319 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753322 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753324 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753327 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.753357 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753329 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753333 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753336 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753338 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753340 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753343 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753345 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753348 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753351 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753353 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753355 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753358 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753360 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753363 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753366 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753369 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753372 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753374 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753377 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753380 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.753919 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753382 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753386 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753390 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753395 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753398 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753401 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753404 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753406 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753408 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753411 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753414 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753416 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753418 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753422 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753425 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753427 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753430 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753432 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753435 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.754380 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753438 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753440 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753442 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753445 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753448 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753450 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753453 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753466 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753468 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753471 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753474 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753476 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753479 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753483 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753485 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753488 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753491 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753493 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753496 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753498 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.754844 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753501 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753503 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753506 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753508 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753511 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753513 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753517 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753520 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753523 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753525 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753528 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753530 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753533 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753536 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753538 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753541 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753543 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753546 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753548 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.755323 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753551 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753553 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753557 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753560 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753563 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.753565 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754347 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754355 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754358 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754361 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754364 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754367 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754369 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754372 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754374 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754377 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754380 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754382 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.755781 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754389 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754392 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754395 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754398 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754400 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754402 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754405 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754407 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754410 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754412 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754415 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754418 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754420 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754423 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754425 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754428 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754430 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754433 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754436 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754438 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.756268 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754441 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754444 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754446 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754449 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754451 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754454 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754456 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754459 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754461 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754464 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754467 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754470 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754472 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754475 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754477 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754480 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754483 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754485 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754488 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754490 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.756747 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754493 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754496 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754498 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754500 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754503 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754505 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754508 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754511 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754513 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754515 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754518 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754520 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754523 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754525 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754529 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754533 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754538 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754541 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754543 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.757241 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754546 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754549 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754551 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754555 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754557 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754563 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754565 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754568 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754570 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754573 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754575 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754578 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754582 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754585 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754588 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754591 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754594 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754597 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.757710 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754599 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754602 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754604 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754606 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754609 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754611 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754614 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754616 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754619 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754622 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754625 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754627 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754630 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754632 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.754635 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755122 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755125 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755128 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755131 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.758163 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755134 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755139 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755142 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755145 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755147 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755150 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755153 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755155 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755158 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755161 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755163 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755166 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755169 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755171 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755174 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755176 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755179 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755181 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755184 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.758623 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755186 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755189 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755194 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755198 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755201 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755204 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755206 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755210 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755214 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755217 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755220 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755223 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755226 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755228 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755231 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755234 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755236 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755239 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755242 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.759091 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755244 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755247 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755249 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755252 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755255 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755257 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755260 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755262 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755265 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755267 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755270 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755272 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755275 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755277 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755280 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755285 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755288 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755291 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755296 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.759535 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755300 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755303 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755306 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755308 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755311 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755313 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755316 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755318 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755321 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755324 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755327 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755329 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755332 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755335 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755337 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755340 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755342 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755345 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755347 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755350 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.759999 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755352 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755355 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755357 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755360 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755362 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755365 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755367 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755371 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755375 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755385 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755388 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755390 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755393 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755395 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755398 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755400 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755403 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755406 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755408 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.760461 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755411 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755413 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755416 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.755418 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757245 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757256 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757263 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757268 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757273 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757276 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757280 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757284 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757288 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757291 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757294 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757298 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757301 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757304 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757307 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757310 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757313 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757317 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757323 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757327 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:41:03.760934 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757330 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757333 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757336 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757340 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757343 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757346 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757349 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757352 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757356 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757358 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757362 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757365 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757370 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757372 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757375 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757378 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757381 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757384 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757388 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757391 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757394 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757397 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757400 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757404 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:41:03.761476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757407 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757410 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757414 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757417 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757420 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757423 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757426 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757429 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757432 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757435 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757439 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757443 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757446 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757449 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757452 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757455 2579 flags.go:64] FLAG: --help="false" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757458 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757461 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757464 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757467 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757471 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757474 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757477 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:41:03.762117 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757480 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757483 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757486 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757489 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757492 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757495 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757498 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757501 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757504 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757506 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757509 2579 flags.go:64] FLAG: --lock-file="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757512 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757515 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757518 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757525 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757527 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757530 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757533 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757536 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757540 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757543 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757545 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757550 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757553 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757556 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 19:41:03.762671 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757559 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757562 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757565 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757568 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757571 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757574 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757577 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757584 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757587 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757589 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757593 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757596 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757598 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757602 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757605 2579 flags.go:64] FLAG: --port="10250" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757608 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757610 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01e47e2e893ca1655" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757614 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757617 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757620 2579 flags.go:64] FLAG: --register-node="true" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757622 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757626 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757629 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757632 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757635 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757639 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:41:03.763276 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757642 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757646 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757649 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757651 2579 flags.go:64] FLAG: --runonce="false" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757654 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757657 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757660 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757663 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757666 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757677 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757680 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757683 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757686 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757689 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757692 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757705 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757709 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757712 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757715 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757722 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757726 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757729 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757733 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757736 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757741 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:41:03.763902 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757744 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757747 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757750 2579 flags.go:64] FLAG: --v="2" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757754 2579 flags.go:64] FLAG: --version="false" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757758 2579 flags.go:64] FLAG: --vmodule="" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757762 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.757766 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757909 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757914 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757917 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757920 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757923 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757925 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757928 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757931 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757933 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757936 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757938 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757941 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757943 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757946 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.764480 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757948 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757951 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757953 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757956 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757958 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757963 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757965 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757968 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757970 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757973 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757977 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757979 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757982 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757984 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757987 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757989 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757991 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757996 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.757999 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.765257 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758002 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758004 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758007 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758010 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758012 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758015 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758018 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758020 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758023 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758026 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758029 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758032 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758035 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758038 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758040 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758043 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758045 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758049 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758052 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.765954 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758054 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758057 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758059 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758064 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758066 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758069 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758072 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758074 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758077 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758079 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758082 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758085 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758087 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758090 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758093 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758095 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758098 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758100 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758103 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.766813 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758105 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758108 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758110 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758113 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758115 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758118 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758120 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758123 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758125 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758128 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758130 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758134 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758136 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758139 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758141 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758144 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758148 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758150 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758153 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.767616 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758158 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758161 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758164 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758167 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758169 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758172 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758176 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758180 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758184 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758187 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758190 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758192 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758195 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758198 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.758224 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:03.768471 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758267 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758271 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758274 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758277 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758280 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758282 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758285 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758288 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758292 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758295 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758297 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758300 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758303 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758305 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758310 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758312 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758315 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758317 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758320 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758322 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.769228 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758325 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758327 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758330 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758333 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758335 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758338 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758341 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758344 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758346 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758349 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758351 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758354 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758356 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758359 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758361 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758364 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758366 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758369 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758371 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758374 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.770111 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758378 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758380 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758383 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758385 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758388 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758392 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758398 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758401 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758404 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758408 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758411 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758414 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758419 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758422 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758424 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758427 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758430 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758432 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.770662 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758435 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758438 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758441 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758444 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758446 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758448 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758451 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758453 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758456 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758458 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758461 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758463 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758466 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758469 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758472 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758474 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758477 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758479 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758485 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.771189 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758489 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758492 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758494 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758497 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758499 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758502 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758504 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758507 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758509 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758512 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758514 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758519 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758522 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758525 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758530 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758533 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758535 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758538 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.771760 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758540 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758543 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758547 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758550 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758555 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758557 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758561 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.758563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.768059 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.768082 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.768106 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.768114 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768202 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768209 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768214 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768220 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768225 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768229 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768235 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.772245 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768239 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768244 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768248 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768252 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768257 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768261 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768265 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768269 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768274 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768279 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768284 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768288 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768292 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768297 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768301 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768305 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768310 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768314 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768318 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768323 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.772802 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768329 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768334 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768339 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768343 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768347 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768352 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768357 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768361 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768366 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768370 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768374 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768379 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768383 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768387 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768391 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768396 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768404 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768409 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.773313 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768414 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768418 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768422 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768427 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768431 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768435 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768440 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768444 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768448 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768453 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768457 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768461 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768466 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768470 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768474 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768479 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768483 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768487 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768492 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768496 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.773778 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768501 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768505 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768509 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768519 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768523 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768527 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768532 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768536 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768540 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768545 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768549 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768553 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768559 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768563 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768567 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768571 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768576 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768580 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.774404 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768584 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768589 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768598 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768602 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768609 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768617 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768623 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768628 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768633 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768638 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768642 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768647 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768651 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768655 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768659 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768664 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768669 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768673 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.775200 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768678 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768682 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.768738 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768986 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.768996 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769001 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769006 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769021 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769025 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769031 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769035 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769040 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769045 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769049 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769053 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.775763 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769058 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769062 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769067 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769073 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769080 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769085 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769090 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769094 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769098 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769103 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769107 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769112 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769120 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769124 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769128 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769132 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769136 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769141 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.776149 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769145 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769149 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769153 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769157 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769162 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769166 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769171 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769176 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769180 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769184 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769189 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769193 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769198 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769203 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769207 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769211 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769215 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769219 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769224 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769228 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.776672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769232 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769236 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769239 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769243 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769247 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769251 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769255 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769259 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769264 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769268 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769272 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769276 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769280 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769285 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769289 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769293 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769297 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769301 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769305 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769312 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.777158 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769316 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769320 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769325 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769329 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769333 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769337 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769341 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769346 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769350 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769354 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769358 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769362 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769366 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769370 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769375 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769379 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769383 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769386 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769391 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.777850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769396 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769404 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769408 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769412 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769419 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769425 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769429 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769433 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769438 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769442 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769446 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769451 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769456 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769460 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.778291 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.769500 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769552 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769559 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769563 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769569 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769576 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769582 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769587 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769591 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769596 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769601 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769607 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769611 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769616 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769621 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769625 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769630 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769634 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769642 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:03.778672 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769646 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769651 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769655 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769659 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769663 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769675 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769679 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769684 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769689 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769693 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769714 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769720 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769726 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769730 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769734 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769738 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769743 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769747 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769752 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:03.779177 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769756 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769760 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769764 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769769 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769773 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769777 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769781 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769785 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769789 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769794 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769798 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769802 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769806 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769810 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769814 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769819 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769823 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769827 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769831 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769836 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:03.779655 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769840 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769845 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769849 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769854 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769858 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769869 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769873 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769877 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769882 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769886 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769890 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769894 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769898 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769902 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769907 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769911 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769915 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769919 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769923 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:03.780204 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769928 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769932 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769936 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769940 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769944 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769949 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769953 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769960 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769965 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769969 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769974 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769978 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769983 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769987 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769992 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.769996 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770000 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770005 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770010 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:03.781019 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770014 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770018 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770022 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770027 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770031 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770035 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770045 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:03.770050 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.770157 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.776336 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.777365 2579 server.go:1014] "Starting client certificate rotation" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.777466 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.777516 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.779474 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:41:03.781505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.780569 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:41:03.790974 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.790956 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:41:03.794531 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.794515 2579 log.go:25] "Validated CRI v1 image API" Oct 30 19:41:03.795376 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.795342 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:41:03.801576 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.801558 2579 fs.go:135] Filesystem UUIDs: map[60386cfd-a6b2-4c7d-8596-23556d32c656:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b7fe18c1-03e7-4ef2-90fe-2e2416b6a3a0:/dev/nvme0n1p3] Oct 30 19:41:03.801623 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.801576 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:41:03.808487 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.808362 2579 manager.go:217] Machine: {Timestamp:2026-10-30 19:41:03.806961215 +0000 UTC m=+0.349076252 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076390 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2ecd5319b4579e5b28e173fdc453 SystemUUID:ec2d2ecd-5319-b457-9e5b-28e173fdc453 BootID:95106406-f6a3-46a0-9648-d8e64be07aac Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:02:49:f3:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:02:49:f3:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:c4:37:db:e5:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:41:03.809152 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.809141 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:41:03.809240 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.809227 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:41:03.812016 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.811994 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:41:03.812142 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.812018 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-41.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:41:03.812796 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.812786 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:41:03.812827 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.812798 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:41:03.812854 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.812828 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:41:03.812854 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.812841 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:41:03.814152 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.814142 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:41:03.821186 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.821173 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:41:03.821295 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.821286 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:41:03.822108 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822098 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:41:03.822154 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822111 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:41:03.822154 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822125 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:41:03.822154 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822134 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:41:03.822154 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822142 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:41:03.822471 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.822460 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:41:03.824764 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.824746 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:41:03.831958 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.831942 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:41:03.833130 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833118 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:41:03.833168 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833148 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:41:03.833313 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833302 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:41:03.833347 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833324 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:41:03.833347 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833334 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:41:03.833399 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833350 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:41:03.833399 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833365 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:41:03.833399 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833375 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:41:03.833999 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.833990 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:41:03.834036 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.834000 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:41:03.834036 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.834007 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:41:03.834036 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.834014 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:41:03.834036 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.834028 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:41:03.834186 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.834039 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:41:03.836925 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.836910 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:41:03.836968 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.836932 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:41:03.840754 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.840741 2579 server.go:1263] "Started kubelet" Oct 30 19:41:03.840937 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.840818 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:41:03.841032 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.840833 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:41:03.841113 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.841098 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:41:03.841576 ip-10-0-130-41 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:41:03.842417 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.842394 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:41:03.844174 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.844153 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:41:03.849066 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.849045 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:41:03.849489 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.849466 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:41:03.849651 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.849638 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:41:03.850084 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.850064 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:03.850536 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850515 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:41:03.850611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850516 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:41:03.850611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850560 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:41:03.850737 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850647 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:41:03.850737 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850658 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:41:03.850815 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850801 2579 factory.go:55] Registering systemd factory Oct 30 19:41:03.850860 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.850818 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:41:03.851028 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.851015 2579 factory.go:153] Registering CRI-O factory Oct 30 19:41:03.851083 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.851033 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 19:41:03.851137 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.851091 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:41:03.851137 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.851110 2579 factory.go:103] Registering Raw factory Oct 30 19:41:03.851137 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.851125 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 19:41:03.852236 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.852220 2579 manager.go:319] Starting recovery of all containers Oct 30 19:41:03.862917 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.862898 2579 manager.go:324] Recovery completed Oct 30 19:41:03.867528 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.867157 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:03.868577 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.868554 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:03.868663 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.868595 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:03.868663 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.868610 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:03.869199 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869180 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:03.869257 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869181 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:41:03.869257 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869231 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:41:03.869257 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869254 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:41:03.869832 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869817 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:03.869911 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869849 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:03.869911 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.869870 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:03.872045 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.872031 2579 policy_none.go:50] "Start" Oct 30 19:41:03.872115 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.872048 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:41:03.872115 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.872061 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:41:03.873823 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.873810 2579 policy_none.go:44] "Start" Oct 30 19:41:03.898407 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.898374 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:41:03.902908 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.902890 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.902966 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.902978 2579 server.go:87] "Starting device plugin registration server" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.903220 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.903233 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.904116 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.904197 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.904204 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.905249 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.905292 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.919809 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.920948 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.920973 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.920992 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:41:03.928234 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.921031 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:41:03.991718 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:03.991679 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-41.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:41:03.992735 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:03.991669 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-41.us-west-1.compute.internal.18e366089f39a89d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-41.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-41.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:03.840716957 +0000 UTC m=+0.382831989,LastTimestamp:2026-10-30 19:41:03.840716957 +0000 UTC m=+0.382831989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-41.us-west-1.compute.internal,}" Oct 30 19:41:04.003900 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.003857 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.004563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.004549 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.004632 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.004575 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.004632 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.004588 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.004632 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.004610 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.021389 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.021372 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal"] Oct 30 19:41:04.021460 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.021424 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.022189 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.022175 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.022252 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.022203 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.022252 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.022213 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.024292 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024280 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.024446 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.024481 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024460 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.024990 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024972 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.025058 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024996 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.025058 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.024978 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.025058 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.025041 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.025058 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.025005 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.025058 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.025056 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.026981 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.026967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.027022 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.026992 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.027603 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.027590 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.027657 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.027611 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.027657 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.027621 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.040237 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.040217 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-41.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:41:04.052580 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.052559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.052665 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.052589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.052665 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.052611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.067233 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.065175 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.068275 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.068257 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.068826 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.068810 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:41:04.068873 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.068866 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cc9dh" Oct 30 19:41:04.113963 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.113938 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-41.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:41:04.113963 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.113941 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:41:04.114086 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.113993 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 19:41:04.153250 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.153314 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.153314 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.153314 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.153440 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.153440 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.153371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.161933 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.161913 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:41:04.162054 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.161980 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-41.us-west-1.compute.internal.18e36608a0e2c6b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-41.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-41.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-41.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:03.868577465 +0000 UTC m=+0.410692503,LastTimestamp:2026-10-30 19:41:03.868577465 +0000 UTC m=+0.410692503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-41.us-west-1.compute.internal,}" Oct 30 19:41:04.201127 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.201102 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.345504 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.345445 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.369841 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.369823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.372980 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.372965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.401282 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.401265 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:04.402052 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.402039 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:04.402099 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.402067 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:04.402099 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.402078 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:04.402165 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.402104 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.459857 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.459841 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:04.459928 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.459860 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-41.us-west-1.compute.internal\": node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:04.595246 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.595224 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:04.695770 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.695749 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:04.776824 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:04.776806 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:41:04.777225 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.776945 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a8a25ac9f0dfc4e67bf96bf3051bc6f6-5b74f0dd39b75e91.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-41.us-west-1.compute.internal.18e36608a0e35bbc\": read tcp 10.0.130.41:50164->18.227.17.225:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-41.us-west-1.compute.internal.18e36608a0e35bbc default 4851 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-41.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-41.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-41.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:03 +0000 UTC,LastTimestamp:2026-10-30 19:41:04.004591711 +0000 UTC m=+0.546706742,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-41.us-west-1.compute.internal,}" Oct 30 19:41:04.796069 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.796052 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:04.896764 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.896693 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:04.997516 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:04.997499 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:05.075517 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.075483 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:03 +0000 UTC" deadline="2028-07-16 13:04:39.07243498 +0000 UTC" Oct 30 19:41:05.075599 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.075518 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14993h23m33.996925053s" Oct 30 19:41:05.098329 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:05.098311 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:05.144781 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.144764 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:05.150829 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.150791 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:05.212404 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.212383 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:05.214230 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.214218 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:05.269551 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.269533 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:05.437100 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:05.437060 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a99c2ea9ac9c2ae64a2a379a47ab1fe.slice/crio-1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a WatchSource:0}: Error finding container 1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a: Status 404 returned error can't find the container with id 1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a Oct 30 19:41:05.437538 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:05.437509 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf763e8cc3376af1056e0b1dfc87c5b.slice/crio-f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68 WatchSource:0}: Error finding container f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68: Status 404 returned error can't find the container with id f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68 Oct 30 19:41:05.442280 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.442268 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:41:05.522886 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.522870 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:05.594609 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.594591 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:05.744758 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.744689 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:05.824070 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.824051 2579 apiserver.go:52] "Watching apiserver" Oct 30 19:41:05.850309 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.850293 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:41:05.924832 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.924792 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerStarted","Data":"1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a"} Oct 30 19:41:05.925878 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.925857 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"baf763e8cc3376af1056e0b1dfc87c5b","Type":"ContainerStarted","Data":"f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68"} Oct 30 19:41:05.928039 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.928025 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:41:05.928300 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.928285 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hkb4l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-lg47k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2","openshift-multus/multus-wbfff","openshift-network-diagnostics/network-check-target-tdkh8","openshift-ovn-kubernetes/ovnkube-node-g8v6z","kube-system/global-pull-secret-syncer-jqblf","kube-system/konnectivity-agent-h2ktl","kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-nvffm","openshift-image-registry/node-ca-jk2pd","openshift-multus/network-metrics-daemon-4tzpr","openshift-network-operator/iptables-alerter-j2p4l"] Oct 30 19:41:05.932166 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.932150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.934334 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.934316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:05.934423 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:05.934392 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:05.936551 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.936536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.938714 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.938684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.940836 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.940819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.944882 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.944864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:05.944977 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:05.944939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:05.947758 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.947743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:05.950004 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.949988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:05.952093 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.952076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:05.954284 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.954263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:05.954378 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:05.954332 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:05.956448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.956430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:05.958721 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.958690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:05.961690 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.961792 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.961792 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.961792 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.961943 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.961943 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.961943 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-629cq\" (UniqueName: \"kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:05.962090 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.962090 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.961972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.962090 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:05.962090 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjf69\" (UniqueName: \"kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtf7\" (UniqueName: \"kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.962331 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8q8\" (UniqueName: \"kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963076 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllq9\" (UniqueName: \"kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.962979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mg7d\" (UniqueName: \"kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zgz\" (UniqueName: \"kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:05.963660 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:05.964131 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.964131 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.964131 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:05.964131 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.963538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:05.985595 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.985579 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:41:05.985814 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.985796 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:41:05.985814 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.985802 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:41:05.985959 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.985855 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:41:05.986016 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.985969 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:41:05.986172 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.986158 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:41:05.986260 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.986248 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\"" Oct 30 19:41:05.989882 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.989868 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\"" Oct 30 19:41:05.991330 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.991316 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:41:05.991659 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.991647 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:41:05.991979 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.991967 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:41:05.994279 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.994266 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:41:05.994324 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:05.994309 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.034726 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.034668 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:41:06.034802 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.034725 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mnvns" Oct 30 19:41:06.051603 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.051589 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:41:06.064405 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064462 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064462 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064462 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064498 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064563 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064745 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.064793 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064836 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.064967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.065152 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.064982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629cq\" (UniqueName: \"kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.065194 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.065227 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065261 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065250 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065296 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.065296 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065296 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.065430 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.065430 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065430 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:06.065430 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.065430 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.065611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.065611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsz49\" (UniqueName: \"kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjf69\" (UniqueName: \"kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.065778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.065794 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.065846 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.565833232 +0000 UTC m=+3.107948283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.065992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtf7\" (UniqueName: \"kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066153 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066242 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.066765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066428 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066469 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066509 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8q8\" (UniqueName: \"kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066866 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllq9\" (UniqueName: \"kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.067338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.066909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067065 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchwj\" (UniqueName: \"kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mg7d\" (UniqueName: \"kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067194 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067296 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068053 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067546 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zgz\" (UniqueName: \"kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067695 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.068605 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.067758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.070454 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.070437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.087157 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087136 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.087266 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087249 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:41:06.087266 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087250 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:41:06.087366 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087293 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:41:06.087366 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087310 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.087366 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087336 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:41:06.087366 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087339 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.087366 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087353 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087380 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087382 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087403 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087427 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087460 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087477 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087486 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:41:06.087586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087584 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\"" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087659 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\"" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087808 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\"" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087827 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\"" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087874 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:41:06.087932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087929 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\"" Oct 30 19:41:06.088116 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.087974 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\"" Oct 30 19:41:06.088116 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.088022 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\"" Oct 30 19:41:06.096561 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.096543 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.097247 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.097228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.098175 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.098145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:06.125806 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.125786 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:06.125806 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.125807 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:06.125913 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.125816 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:06.125913 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.125871 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.625857754 +0000 UTC m=+3.167972773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:06.126840 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.126824 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtf7\" (UniqueName: \"kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.127051 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.127035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629cq\" (UniqueName: \"kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.127697 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.127683 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjf69\" (UniqueName: \"kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.130664 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.130643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllq9\" (UniqueName: \"kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.130892 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.130876 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8q8\" (UniqueName: \"kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.131199 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.131181 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mg7d\" (UniqueName: \"kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.131263 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.131247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zgz\" (UniqueName: \"kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:06.168581 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.168664 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.168664 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168664 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168658 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168685 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168694 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchwj\" (UniqueName: \"kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168764 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.168839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168958 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168989 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.168993 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsz49\" (UniqueName: \"kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169170 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169287 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.169448 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.169338 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:06.170325 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.169362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.170325 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.169382 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.669368858 +0000 UTC m=+3.211483892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:06.170846 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.170828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.170904 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.170860 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.231932 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.231911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchwj\" (UniqueName: \"kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.233922 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.233900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsz49\" (UniqueName: \"kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.266211 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.266194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbfff" Oct 30 19:41:06.270084 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.270064 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:06.275850 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.275830 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1124e17_db0a_42d9_8be6_82db4177cf24.slice/crio-90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74 WatchSource:0}: Error finding container 90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74: Status 404 returned error can't find the container with id 90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74 Oct 30 19:41:06.277778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.277756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:06.277865 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.277794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:06.279475 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.279424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:06.279694 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.279675 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0525652_ce3f_4898_9be8_066de957058e.slice/crio-955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70 WatchSource:0}: Error finding container 955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70: Status 404 returned error can't find the container with id 955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70 Oct 30 19:41:06.283996 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.283981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:06.285573 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.285529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:06.290976 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.290955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:06.291266 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.291248 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ef5a23_6398_4f49_b7f3_09f18b3473ca.slice/crio-9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910 WatchSource:0}: Error finding container 9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910: Status 404 returned error can't find the container with id 9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910 Oct 30 19:41:06.291779 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.291758 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14438f5_ecd7_4a84_9bb4_7648b87bd075.slice/crio-cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd WatchSource:0}: Error finding container cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd: Status 404 returned error can't find the container with id cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd Oct 30 19:41:06.292618 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.292604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:06.298731 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.298694 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5c9fe7_ca03_478a_85ad_1a6e4afb4327.slice/crio-a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664 WatchSource:0}: Error finding container a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664: Status 404 returned error can't find the container with id a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664 Oct 30 19:41:06.300261 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.300238 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d03ece_45f6_4355_8414_729003908b01.slice/crio-d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35 WatchSource:0}: Error finding container d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35: Status 404 returned error can't find the container with id d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35 Oct 30 19:41:06.306525 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.306484 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe11c10_4c93_48fc_a213_0faaf29b5d11.slice/crio-66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91 WatchSource:0}: Error finding container 66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91: Status 404 returned error can't find the container with id 66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91 Oct 30 19:41:06.309852 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:06.309829 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e60683e_1ea5_4951_aae1_c76f68ef297a.slice/crio-f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce WatchSource:0}: Error finding container f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce: Status 404 returned error can't find the container with id f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce Oct 30 19:41:06.572490 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.572420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:06.572621 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.572553 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:06.572621 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.572597 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:07.572585879 +0000 UTC m=+4.114700911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:06.673555 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.673522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:06.673753 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.673560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:06.673753 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673718 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:06.673753 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673737 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:06.673753 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673750 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:06.673978 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673801 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:07.673784312 +0000 UTC m=+4.215899351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:06.673978 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673875 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:06.673978 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:06.673910 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:07.67389923 +0000 UTC m=+4.216014264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:06.929091 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.929054 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvffm" event={"ID":"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7","Type":"ContainerStarted","Data":"bba920bb57a2a158107df46e0dd5c4b7f710cc6ef058517bea972c6e5bb0e9aa"} Oct 30 19:41:06.931688 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.930964 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j2p4l" event={"ID":"d1d03ece-45f6-4355-8414-729003908b01","Type":"ContainerStarted","Data":"d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35"} Oct 30 19:41:06.932321 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.932284 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70"} Oct 30 19:41:06.933612 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.933589 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jk2pd" event={"ID":"6e60683e-1ea5-4951-aae1-c76f68ef297a","Type":"ContainerStarted","Data":"f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce"} Oct 30 19:41:06.934985 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.934962 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" event={"ID":"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327","Type":"ContainerStarted","Data":"a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664"} Oct 30 19:41:06.936284 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.936254 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h2ktl" event={"ID":"e14438f5-ecd7-4a84-9bb4-7648b87bd075","Type":"ContainerStarted","Data":"cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd"} Oct 30 19:41:06.937644 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.937616 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkb4l" event={"ID":"80ef5a23-6398-4f49-b7f3-09f18b3473ca","Type":"ContainerStarted","Data":"9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910"} Oct 30 19:41:06.939571 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.939529 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbfff" event={"ID":"e1124e17-db0a-42d9-8be6-82db4177cf24","Type":"ContainerStarted","Data":"90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74"} Oct 30 19:41:06.940956 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:06.940929 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerStarted","Data":"66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91"} Oct 30 19:41:07.038584 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.038482 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:05 +0000 UTC" deadline="2028-03-30 23:35:46.90627412 +0000 UTC" Oct 30 19:41:07.038584 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.038526 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12411h54m39.867752241s" Oct 30 19:41:07.583101 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.582928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:07.583101 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.583075 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:07.583304 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.583142 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:09.58312545 +0000 UTC m=+6.125240480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:07.683461 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.683428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:07.683630 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.683473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:07.683714 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683649 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:07.683714 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683668 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:07.683714 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683682 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:07.683850 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:09.683735419 +0000 UTC m=+6.225850439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:07.683850 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683809 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:07.683921 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.683869 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:09.683855918 +0000 UTC m=+6.225970937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:07.924092 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.924062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:07.924305 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.924169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:07.924305 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.924231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:07.924305 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.924281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:07.924460 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:07.924318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:07.924460 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:07.924358 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:08.039399 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:08.039358 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:05 +0000 UTC" deadline="2028-07-26 05:04:23.255404972 +0000 UTC" Oct 30 19:41:08.039399 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:08.039398 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15225h23m15.216010861s" Oct 30 19:41:08.969047 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:08.968813 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a99c2ea9ac9c2ae64a2a379a47ab1fe" containerID="6cb11a7fa7c0552afff11f79f60f2cce9e051ec2fc3c8d3a5fcce090d1a302a2" exitCode=0 Oct 30 19:41:08.969638 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:08.969613 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerDied","Data":"6cb11a7fa7c0552afff11f79f60f2cce9e051ec2fc3c8d3a5fcce090d1a302a2"} Oct 30 19:41:08.977766 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:08.977740 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"baf763e8cc3376af1056e0b1dfc87c5b","Type":"ContainerStarted","Data":"7e1005395cdaa3c37812e9163f78af1e53c9b7ff1d4e441a441f9b1b80b3d271"} Oct 30 19:41:09.206561 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.206310 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" podStartSLOduration=4.206278858 podStartE2EDuration="4.206278858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:09.204368571 +0000 UTC m=+5.746483610" watchObservedRunningTime="2026-10-30 19:41:09.206278858 +0000 UTC m=+5.748393902" Oct 30 19:41:09.601967 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.601939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:09.602114 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.602097 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:09.602184 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.602157 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:13.60213965 +0000 UTC m=+10.144254686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:09.702911 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.702870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:09.703066 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.702929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:09.703135 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703068 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:09.703135 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703090 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:09.703135 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703102 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:09.703281 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703163 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:13.703144766 +0000 UTC m=+10.245259786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:09.703281 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703239 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:09.703281 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.703275 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:13.703263993 +0000 UTC m=+10.245379026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.924024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.924065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.924159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.924256 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.924299 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:09.924548 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:09.924371 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:09.989629 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.989569 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerStarted","Data":"70a923ff705cd5d0f3b54112c28e1001f83fef1dfb96917d32afcf359bcbfc6f"} Oct 30 19:41:09.994518 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.993994 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="a0988406e69cca93c3b9f1cb8b2196c6e36d5ca421c77e0678f12b0d7ef33851" exitCode=0 Oct 30 19:41:09.994518 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:09.994105 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"a0988406e69cca93c3b9f1cb8b2196c6e36d5ca421c77e0678f12b0d7ef33851"} Oct 30 19:41:10.232531 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:10.231608 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" podStartSLOduration=5.231588829 podStartE2EDuration="5.231588829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:10.109851854 +0000 UTC m=+6.651966896" watchObservedRunningTime="2026-10-30 19:41:10.231588829 +0000 UTC m=+6.773703871" Oct 30 19:41:11.001622 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:11.001590 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h2ktl" event={"ID":"e14438f5-ecd7-4a84-9bb4-7648b87bd075","Type":"ContainerStarted","Data":"62a58620c53d3a22fa7be3a93393b5efc4e9038f4d814f6c151a26cef4838fec"} Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:11.924201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:11.924224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:11.924333 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:11.924375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:11.924469 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:11.924688 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:11.924608 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:13.014602 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.014562 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" event={"ID":"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327","Type":"ContainerStarted","Data":"4a2605513bac05e169511c558ebfb02d73079f3e78e855b3fb6183650113a863"} Oct 30 19:41:13.016797 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.016772 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jk2pd" event={"ID":"6e60683e-1ea5-4951-aae1-c76f68ef297a","Type":"ContainerStarted","Data":"777b2894cfeaeb47a4fab67ab1ae6e04c6f6bc7a10b95a47ea3bf22e26b86d9d"} Oct 30 19:41:13.137077 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.137019 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h2ktl" podStartSLOduration=4.86860375 podStartE2EDuration="9.137000595s" totalImagesPullingTime="4.268396845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:11.117035319 +0000 UTC m=+7.659150356" watchObservedRunningTime="2026-10-30 19:41:13.137000595 +0000 UTC m=+9.679115638" Oct 30 19:41:13.643098 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.643060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:13.643282 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.643246 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:13.643354 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.643302 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:21.643290214 +0000 UTC m=+18.185405233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:13.743656 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.743612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:13.743853 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.743680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:13.743914 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.743893 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:13.743974 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.743917 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:13.743974 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.743932 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:13.744076 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.743996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:21.743976316 +0000 UTC m=+18.286091358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:13.744434 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.744415 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:13.744524 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.744469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:21.744453475 +0000 UTC m=+18.286568508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:13.869541 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.868969 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:13.870155 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.869971 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:13.924517 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.924019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:13.924517 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.924152 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:13.924517 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.924287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:13.924517 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.924441 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:13.924517 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.924512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:13.924781 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:13.924580 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:13.998721 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:13.997009 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jk2pd" podStartSLOduration=4.549044022 podStartE2EDuration="9.996988742s" totalImagesPullingTime="5.44794472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:13.138778664 +0000 UTC m=+9.680893706" watchObservedRunningTime="2026-10-30 19:41:13.996988742 +0000 UTC m=+10.539103784" Oct 30 19:41:14.520845 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:14.520808 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:41:14.917801 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:14.917628 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:41:14.520826173Z","UUID":"cf12acff-4f8f-42b6-aee7-e8ec0bb49fdc","Handler":null,"Name":"","Endpoint":""} Oct 30 19:41:14.919889 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:14.919786 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:41:14.919889 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:14.919856 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:41:15.023627 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.023410 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" event={"ID":"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327","Type":"ContainerStarted","Data":"cda9570a712ade51b3f64a8ee43ce1d9efa3165eff05ce104d37e9546637b785"} Oct 30 19:41:15.025483 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.025452 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkb4l" event={"ID":"80ef5a23-6398-4f49-b7f3-09f18b3473ca","Type":"ContainerStarted","Data":"5e9d06b8d8b77548b0bcd1b88892af1f86608429e4ecbf8381f771e0138de096"} Oct 30 19:41:15.027238 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.027190 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbfff" event={"ID":"e1124e17-db0a-42d9-8be6-82db4177cf24","Type":"ContainerStarted","Data":"6830983aabd73fa4a400044b99813271e2ce2eb9466150de255eac3aee893d60"} Oct 30 19:41:15.029481 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.029450 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="106d4ee8c6a60d552adf57bd709577c35f701175fee311aed9faa5367303dbcd" exitCode=0 Oct 30 19:41:15.029579 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.029495 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"106d4ee8c6a60d552adf57bd709577c35f701175fee311aed9faa5367303dbcd"} Oct 30 19:41:15.263777 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.263660 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hkb4l" podStartSLOduration=2.73412851 podStartE2EDuration="11.263642005s" totalImagesPullingTime="8.529513495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:15.141478719 +0000 UTC m=+11.683593763" watchObservedRunningTime="2026-10-30 19:41:15.263642005 +0000 UTC m=+11.805757047" Oct 30 19:41:15.264042 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.264006 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wbfff" podStartSLOduration=3.050488107 podStartE2EDuration="11.264001009s" totalImagesPullingTime="8.213512902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:15.263408336 +0000 UTC m=+11.805523378" watchObservedRunningTime="2026-10-30 19:41:15.264001009 +0000 UTC m=+11.806116051" Oct 30 19:41:15.923753 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.923720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:15.924216 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:15.923860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:15.924305 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.924274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:15.924442 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:15.924400 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:15.924586 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:15.924505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:15.924643 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:15.924599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:17.036203 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.036169 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="333faff876a057bbf4cdb545115e821a3d94f525f6c3e16b184381ef00d48739" exitCode=0 Oct 30 19:41:17.036775 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.036252 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"333faff876a057bbf4cdb545115e821a3d94f525f6c3e16b184381ef00d48739"} Oct 30 19:41:17.037860 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.037840 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j2p4l" event={"ID":"d1d03ece-45f6-4355-8414-729003908b01","Type":"ContainerStarted","Data":"8539101ecbb291355421b797ccc3106e8c27f9675be8ce8bf913b44e6bdbc80a"} Oct 30 19:41:17.040569 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.040547 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" event={"ID":"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327","Type":"ContainerStarted","Data":"7d8f13e6608240986d055344521c9bdb65171325ea5f059e87e912a94e4dd4b5"} Oct 30 19:41:17.516211 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.516152 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j2p4l" podStartSLOduration=5.238302103 podStartE2EDuration="13.516134411s" totalImagesPullingTime="8.277832308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.374324394 +0000 UTC m=+13.916439434" watchObservedRunningTime="2026-10-30 19:41:17.516134411 +0000 UTC m=+14.058249453" Oct 30 19:41:17.516444 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.516416 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" podStartSLOduration=4.153903568 podStartE2EDuration="13.516410028s" totalImagesPullingTime="9.36250646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.516030433 +0000 UTC m=+14.058145476" watchObservedRunningTime="2026-10-30 19:41:17.516410028 +0000 UTC m=+14.058525069" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.922049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.922068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:17.922158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:17.922169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:17.922242 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:17.922669 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:17.922325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:18.045172 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:18.045137 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvffm" event={"ID":"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7","Type":"ContainerStarted","Data":"6425c580e3631ce3dcb4a232c0f3f85c268be22e51b07378149cc1f1b43581b8"} Oct 30 19:41:18.174406 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:18.174315 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nvffm" podStartSLOduration=3.502789333 podStartE2EDuration="14.174301012s" totalImagesPullingTime="10.671511679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:18.173934257 +0000 UTC m=+14.716049299" watchObservedRunningTime="2026-10-30 19:41:18.174301012 +0000 UTC m=+14.716416053" Oct 30 19:41:19.050235 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:19.050028 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="efc23853816df6694989f2c602cf68cb7d6bccab2d5bf8a747d0d3a060362f34" exitCode=0 Oct 30 19:41:19.050796 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:19.050125 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"efc23853816df6694989f2c602cf68cb7d6bccab2d5bf8a747d0d3a060362f34"} Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:19.921520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:19.921559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:19.921633 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:19.921671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:19.921798 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:19.922732 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:19.921890 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:21.705318 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.705191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:21.705745 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.705321 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:21.705745 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.705371 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:37.705357621 +0000 UTC m=+34.247472644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:41:21.805540 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.805517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:21.805638 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.805548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:21.805716 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.805687 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:41:21.805785 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.805721 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:41:21.805785 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.805730 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jb6fk for pod openshift-network-diagnostics/network-check-target-tdkh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:21.805785 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.805770 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk podName:dd3626c0-00b5-4039-a06c-9537f72283bc nodeName:}" failed. No retries permitted until 2026-10-30 19:41:37.805757794 +0000 UTC m=+34.347872813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jb6fk" (UniqueName: "kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk") pod "network-check-target-tdkh8" (UID: "dd3626c0-00b5-4039-a06c-9537f72283bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:41:21.806185 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.806166 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:21.806247 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.806234 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret podName:cdfc0cb5-a33f-488c-974e-75c43659c08a nodeName:}" failed. No retries permitted until 2026-10-30 19:41:37.806217145 +0000 UTC m=+34.348332167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret") pod "global-pull-secret-syncer-jqblf" (UID: "cdfc0cb5-a33f-488c-974e-75c43659c08a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:41:21.921954 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.921929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:21.922095 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.921960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:21.922095 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:21.921929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:21.922095 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.922056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:21.922191 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.922155 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:21.922292 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:21.922273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:22.058839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.058767 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="3c9f162db630f315c41ef23913c62c42a2d859722f5622c418c77fbd9e186753" exitCode=0 Oct 30 19:41:22.058839 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.058811 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"3c9f162db630f315c41ef23913c62c42a2d859722f5622c418c77fbd9e186753"} Oct 30 19:41:22.061941 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.061920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"886fc63a548c170e5c0b077d4c5119169fd7b52c4d839092710c525e5630cbd2"} Oct 30 19:41:22.061991 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.061975 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"c25b78d84334e94da759614b3cfd578dd9b905527a698c929c2c4e0829ac8609"} Oct 30 19:41:22.062020 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.062010 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"b3d6f49b1307fea11737bc237cc86de5211d5fa283da5e6dfccdfbf43abe3635"} Oct 30 19:41:22.062052 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.062040 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"f765cb2d73c1fcf28b562191a61cc2522eda53e589b6360f0e9fc2659be8390e"} Oct 30 19:41:22.062090 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.062067 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"3b3c27a8b9bd592468b0971b097f563233a2879836e954ce6336c73598d358dd"} Oct 30 19:41:22.062121 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:22.062108 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"e2b700eaf6aba438c04c7bcde57481e75dfc443f1b7bf59818f5062628deb68b"} Oct 30 19:41:23.066458 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:23.066428 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="5b989f07f65246eafbf588ca197f3bd99f912e92a22a466dc912a44dd2b57bd7" exitCode=0 Oct 30 19:41:23.067066 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:23.066484 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerDied","Data":"5b989f07f65246eafbf588ca197f3bd99f912e92a22a466dc912a44dd2b57bd7"} Oct 30 19:41:23.922265 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:23.922242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:23.922401 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:23.922375 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:23.922484 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:23.922421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:23.922539 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:23.922506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:23.922659 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:23.922618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:23.922740 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:23.922721 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:24.071380 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:24.071349 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg47k" event={"ID":"efe11c10-4c93-48fc-a213-0faaf29b5d11","Type":"ContainerStarted","Data":"85308e3481ef5ed9eb09ada009134fde9d383081fdaee5b64008a55a86734197"} Oct 30 19:41:24.074414 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:24.074391 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"b275d2d494c06c1e524e5084e0222b2eb10b68d406022d527460b4f3d682367c"} Oct 30 19:41:24.204528 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:24.204487 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lg47k" podStartSLOduration=6.948796085 podStartE2EDuration="20.204474113s" totalImagesPullingTime="13.255678028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:24.203879803 +0000 UTC m=+20.745994844" watchObservedRunningTime="2026-10-30 19:41:24.204474113 +0000 UTC m=+20.746589155" Oct 30 19:41:25.921939 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:25.921908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:25.922411 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:25.921968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:25.922411 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:25.921972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:25.922411 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:25.922056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:25.922411 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:25.922169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:25.922411 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:25.922245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:26.082812 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:26.082775 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" event={"ID":"e0525652-ce3f-4898-9be8-066de957058e","Type":"ContainerStarted","Data":"7ed5274960f80fa1ea673c800d8a2ec738ea316af761b78012f9cfbb20153d23"} Oct 30 19:41:26.083099 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:26.083073 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:26.097687 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:26.097666 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:26.505644 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:26.505600 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" podStartSLOduration=7.29409201 podStartE2EDuration="22.505588137s" totalImagesPullingTime="15.211496127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:26.368949601 +0000 UTC m=+22.911064642" watchObservedRunningTime="2026-10-30 19:41:26.505588137 +0000 UTC m=+23.047703178" Oct 30 19:41:27.084604 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.084579 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:41:27.085032 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.084980 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:27.099778 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.099758 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:27.922004 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.921973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:27.922282 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.922038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:27.922282 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:27.922042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:27.922282 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:27.922157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:27.922282 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:27.922232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:27.922468 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:27.922326 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:28.086579 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.086560 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:41:28.743039 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.743013 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqblf"] Oct 30 19:41:28.743189 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.743100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:28.743230 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:28.743210 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:28.744346 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.744311 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdkh8"] Oct 30 19:41:28.744482 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.744407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:28.744557 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:28.744496 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:28.744994 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.744967 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tzpr"] Oct 30 19:41:28.745062 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:28.745051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:28.745152 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:28.745132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:29.088330 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:29.088273 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:41:29.921226 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:29.921192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:29.921385 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:29.921202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:29.921385 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:29.921313 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:29.921385 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:29.921375 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:30.921676 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:30.921650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:30.922044 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:30.921650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:30.922044 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:30.921785 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:30.922044 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:30.921818 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:31.921753 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:31.921726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:31.922186 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:31.921830 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:32.921692 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:32.921658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:32.921692 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:32.921681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:32.922258 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:32.921795 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdkh8" podUID="dd3626c0-00b5-4039-a06c-9537f72283bc" Oct 30 19:41:32.922258 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:32.921895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqblf" podUID="cdfc0cb5-a33f-488c-974e-75c43659c08a" Oct 30 19:41:33.001942 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.001911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.236295 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.236219 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:33.236469 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.236336 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:41:33.237208 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.237182 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:33.407746 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.407693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.763730 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.763692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.818635 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.818605 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:33.818810 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.818798 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:41:33.829604 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.829586 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:33.922022 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:33.921997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:33.922362 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:33.922115 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tzpr" podUID="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" Oct 30 19:41:34.224820 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.224630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.301201 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.301142 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeReady" Oct 30 19:41:34.301326 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.301273 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:41:34.434650 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.434622 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k8hkx"] Oct 30 19:41:34.437157 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.437143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.446128 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.446111 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-78n2p"] Oct 30 19:41:34.448655 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.448642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.497538 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.497514 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:41:34.497684 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.497578 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:41:34.497684 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.497675 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\"" Oct 30 19:41:34.497857 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.497728 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:41:34.501005 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.500990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.501088 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.501016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rqg\" (UniqueName: \"kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.501088 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.501054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.501155 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.501111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.501248 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.501235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9g76\" (UniqueName: \"kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.501281 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.501261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.511198 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.511175 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8hkx"] Oct 30 19:41:34.581489 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.581426 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78n2p"] Oct 30 19:41:34.598324 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.598302 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:41:34.598914 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.598900 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:41:34.598974 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.598942 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\"" Oct 30 19:41:34.602123 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.602169 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rqg\" (UniqueName: \"kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.602201 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.602201 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.602258 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g76\" (UniqueName: \"kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.602258 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.602350 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:34.602311 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:34.602407 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:34.602375 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:35.102357169 +0000 UTC m=+31.644472204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:34.602560 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.602694 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.602679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.606036 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.606021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.627860 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.627838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.672128 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.672111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rqg\" (UniqueName: \"kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.679226 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.679199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g76\" (UniqueName: \"kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:34.748300 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.748271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:34.921174 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.921149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:34.921303 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.921175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:34.989454 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.989433 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:41:34.992286 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.992271 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:41:34.993210 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.993190 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\"" Oct 30 19:41:34.993210 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.993205 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\"" Oct 30 19:41:34.995987 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:34.993566 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:41:35.006357 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.006332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:35.105373 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.105350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:35.105486 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:35.105473 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:35.105534 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:35.105525 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.105510247 +0000 UTC m=+32.647625286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:35.157060 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.157038 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8hkx"] Oct 30 19:41:35.161541 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:35.161520 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f786ff_dc91_4035_96cc_2e840a7fcae5.slice/crio-ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa WatchSource:0}: Error finding container ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa: Status 404 returned error can't find the container with id ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa Oct 30 19:41:35.921770 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.921740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:35.974863 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.974834 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:41:35.974863 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:35.974840 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\"" Oct 30 19:41:36.104112 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:36.104086 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8hkx" event={"ID":"24f786ff-dc91-4035-96cc-2e840a7fcae5","Type":"ContainerStarted","Data":"ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa"} Oct 30 19:41:36.111360 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:36.111341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:36.111474 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:36.111462 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:36.111525 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:36.111516 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:38.11150169 +0000 UTC m=+34.653616723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:37.724338 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.724304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:37.724727 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:37.724422 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:37.724727 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:37.724472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:42:09.724460273 +0000 UTC m=+66.266575305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:37.825183 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.825150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:37.825183 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.825187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:37.827451 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.827420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:37.827755 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.827735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:37.867908 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.867880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.937522 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.937499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:37.939638 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:37.939623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:38.126852 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:38.126829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:38.126972 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:38.126936 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:38.127011 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:38.126989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:42.126973086 +0000 UTC m=+38.669088111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:38.214020 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:38.213981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.385598 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:38.385569 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdkh8"] Oct 30 19:41:38.405039 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:38.405012 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3626c0_00b5_4039_a06c_9537f72283bc.slice/crio-39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327 WatchSource:0}: Error finding container 39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327: Status 404 returned error can't find the container with id 39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327 Oct 30 19:41:38.426713 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:38.426678 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqblf"] Oct 30 19:41:38.429966 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:38.429940 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfc0cb5_a33f_488c_974e_75c43659c08a.slice/crio-29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77 WatchSource:0}: Error finding container 29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77: Status 404 returned error can't find the container with id 29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77 Oct 30 19:41:38.810267 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:38.810223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.093832 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:39.093762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.112158 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:39.112128 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqblf" event={"ID":"cdfc0cb5-a33f-488c-974e-75c43659c08a","Type":"ContainerStarted","Data":"29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77"} Oct 30 19:41:39.113286 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:39.113263 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdkh8" event={"ID":"dd3626c0-00b5-4039-a06c-9537f72283bc","Type":"ContainerStarted","Data":"39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327"} Oct 30 19:41:40.121662 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.121620 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8hkx" event={"ID":"24f786ff-dc91-4035-96cc-2e840a7fcae5","Type":"ContainerStarted","Data":"91d290ead68b8dadbb99e854a8110f806b85fd5e2fffef7ba839dc82c6fb783a"} Oct 30 19:41:40.248309 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.248251 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k8hkx" podStartSLOduration=2.045527868 podStartE2EDuration="6.248237671s" totalImagesPullingTime="4.202709803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.247908671 +0000 UTC m=+36.790023712" watchObservedRunningTime="2026-10-30 19:41:40.248237671 +0000 UTC m=+36.790352712" Oct 30 19:41:40.722166 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.722131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:40.733303 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.732513 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh"] Oct 30 19:41:40.749476 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.749453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:40.809324 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.809291 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh"] Oct 30 19:41:40.846694 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.846661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qblk\" (UniqueName: \"kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk\") pod \"migrator-66df5d496b-tk7kh\" (UID: \"24b7a886-5347-4494-a757-db73e0177366\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:40.858365 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.858333 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:41:40.858365 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.858341 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:40.858557 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.858379 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v4lmq\"" Oct 30 19:41:40.947972 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.947932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qblk\" (UniqueName: \"kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk\") pod \"migrator-66df5d496b-tk7kh\" (UID: \"24b7a886-5347-4494-a757-db73e0177366\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:40.990418 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:40.990326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.009937 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.009909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qblk\" (UniqueName: \"kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk\") pod \"migrator-66df5d496b-tk7kh\" (UID: \"24b7a886-5347-4494-a757-db73e0177366\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:41.073377 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.073353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:41.236812 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.236772 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w9tj4"] Oct 30 19:41:41.271631 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.271556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.302825 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.302798 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9tj4"] Oct 30 19:41:41.306455 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.306428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.352044 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.352011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.352217 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.352152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.352564 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.352373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqqc\" (UniqueName: \"kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.352564 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.352416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.352564 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.352445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.369204 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.369151 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:41:41.369204 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.369160 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:41:41.369204 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.369181 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:41:41.370463 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.370441 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\"" Oct 30 19:41:41.370463 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.370461 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:41:41.454254 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.454218 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:41:41.455459 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455567 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455567 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqqc\" (UniqueName: \"kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455567 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455567 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455765 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.455682 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:41.455765 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455765 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.455749 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:41.955737995 +0000 UTC m=+38.497853014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:41.455925 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.455788 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:41:41.455925 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.455809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.455925 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.455815 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:41.955808142 +0000 UTC m=+38.497923193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : configmap "kube-rbac-proxy" not found Oct 30 19:41:41.479537 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.479513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.523039 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.522978 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:41:41.529428 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.529401 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqqc\" (UniqueName: \"kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.532132 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.532106 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh"] Oct 30 19:41:41.532897 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.532880 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hpq6h\"" Oct 30 19:41:41.536936 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:41.536910 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b7a886_5347_4494_a757_db73e0177366.slice/crio-d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2 WatchSource:0}: Error finding container d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2: Status 404 returned error can't find the container with id d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2 Oct 30 19:41:41.556420 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.556388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.556509 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.556430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.556546 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.556512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.556546 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.556537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.565964 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.565938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.619424 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.619380 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:41:41.657852 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.657812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.657852 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.657860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.658055 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.657932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.658055 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.657956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.658640 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.658613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.658998 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.658972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.670030 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.670007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:41:41.679596 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.679575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.720877 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.720851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.804068 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.803998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:41.959897 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.959865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.960059 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.959903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:41.960059 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.959997 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:41:41.960059 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.960018 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:41.960059 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.960059 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:42.960047898 +0000 UTC m=+39.502162918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:41.960266 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:41.960076 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:42.96006641 +0000 UTC m=+39.502181446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : configmap "kube-rbac-proxy" not found Oct 30 19:41:41.973888 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:41.973840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.127959 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.127876 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" event={"ID":"24b7a886-5347-4494-a757-db73e0177366","Type":"ContainerStarted","Data":"d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2"} Oct 30 19:41:42.161222 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.161180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:42.161375 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.161343 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:42.161417 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.161399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:50.161385322 +0000 UTC m=+46.703500342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:42.207230 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.207195 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:41:42.214825 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:42.214797 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf076c5_f276_4191_9240_8968d8b20978.slice/crio-2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835 WatchSource:0}: Error finding container 2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835: Status 404 returned error can't find the container with id 2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835 Oct 30 19:41:42.279663 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.279630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.530209 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.530179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.967836 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.967800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:42.968127 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:42.967841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:42.968127 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.967943 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:41:42.968127 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.968025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.968004423 +0000 UTC m=+41.510119464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : configmap "kube-rbac-proxy" not found Oct 30 19:41:42.968127 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.968024 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:42.968127 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:42.968090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.968073828 +0000 UTC m=+41.510188850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:43.129265 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.129229 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g9npz"] Oct 30 19:41:43.153777 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.153751 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:43.153919 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.153786 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" event={"ID":"3bf076c5-f276-4191-9240-8968d8b20978","Type":"ContainerStarted","Data":"2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835"} Oct 30 19:41:43.153919 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.153812 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdkh8" event={"ID":"dd3626c0-00b5-4039-a06c-9537f72283bc","Type":"ContainerStarted","Data":"a30797644c23e5caefef0f729c3199477074dd3f36c15ad67f1933bbc798e47f"} Oct 30 19:41:43.153919 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.153891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.194637 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.194609 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g9npz"] Oct 30 19:41:43.207891 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.207868 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:41:43.210611 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.210587 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:41:43.210730 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.210642 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:41:43.210730 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.210668 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:41:43.256877 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.256784 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tdkh8" podStartSLOduration=35.120898461 podStartE2EDuration="39.256769754s" totalImagesPullingTime="4.135871293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:43.256420066 +0000 UTC m=+39.798535107" watchObservedRunningTime="2026-10-30 19:41:43.256769754 +0000 UTC m=+39.798884797" Oct 30 19:41:43.271216 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.270200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzqr8\" (UniqueName: \"kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.271216 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.270311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.271216 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.270372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.309536 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.309512 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9h48w\"" Oct 30 19:41:43.370886 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.370860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.371034 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.370935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.371034 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.370993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzqr8\" (UniqueName: \"kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.372435 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.372408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.375109 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.374553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.437790 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.437384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzqr8\" (UniqueName: \"kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.478468 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.478440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:43.583458 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.583183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.856171 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.856087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.904505 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:43.904471 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g9npz"] Oct 30 19:41:43.908459 ip-10-0-130-41 kubenswrapper[2579]: W1030 19:41:43.908431 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260bbf80_e9c9_4264_8983_7bf37d39a563.slice/crio-62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87 WatchSource:0}: Error finding container 62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87: Status 404 returned error can't find the container with id 62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87 Oct 30 19:41:44.139367 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:44.139334 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" event={"ID":"260bbf80-e9c9-4264-8983-7bf37d39a563","Type":"ContainerStarted","Data":"62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87"} Oct 30 19:41:44.986164 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:44.986133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:44.986164 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:44.986167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:44.986575 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:44.986266 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:41:44.986575 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:44.986301 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:44.986575 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:44.986342 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.986322006 +0000 UTC m=+45.528437231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : configmap "kube-rbac-proxy" not found Oct 30 19:41:44.986575 ip-10-0-130-41 kubenswrapper[2579]: E1030 19:41:44.986362 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.986352215 +0000 UTC m=+45.528467234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:45.010601 ip-10-0-130-41 kubenswrapper[2579]: I1030 19:41:45.010577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.104352 ip-10-0-130-41 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:41:45.115139 ip-10-0-130-41 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:41:45.115316 ip-10-0-130-41 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:41:45.115363 ip-10-0-130-41 systemd[1]: kubelet.service: Consumed 2.317s CPU time. Oct 30 19:41:45.127271 ip-10-0-130-41 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:41:45.217682 ip-10-0-130-41 kubenswrapper[4524]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:45.217682 ip-10-0-130-41 kubenswrapper[4524]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:41:45.217682 ip-10-0-130-41 kubenswrapper[4524]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:45.217682 ip-10-0-130-41 kubenswrapper[4524]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:41:45.219763 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219640 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.219763 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219763 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219767 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219771 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219774 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219777 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219779 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219782 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219785 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219788 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219791 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219793 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219796 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219798 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219801 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219803 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219806 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219809 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219811 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219814 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.219839 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219816 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219819 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219821 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219824 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219826 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219829 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219838 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219841 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219845 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219847 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219850 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219853 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219856 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219859 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219861 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219864 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219867 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219869 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219872 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219875 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.220309 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219877 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219880 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219882 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219887 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219889 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219892 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219895 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219897 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219900 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219903 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219905 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219908 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219910 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219913 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219917 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219920 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219923 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219925 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219928 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.220832 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219931 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219933 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219936 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219938 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219941 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219943 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219946 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219949 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219951 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219953 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219956 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219959 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219961 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219964 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219967 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219972 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219975 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219979 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219982 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.221321 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219985 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219990 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219993 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.219997 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220000 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220002 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220005 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220007 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220011 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220013 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220016 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220019 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220021 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220024 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220027 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220030 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220033 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220035 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220039 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.221814 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220043 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220046 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220049 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220051 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220054 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220056 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220059 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220061 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220064 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220829 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220837 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220841 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220844 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220848 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220851 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220854 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220857 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220859 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220862 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220864 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.222281 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220867 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220870 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220873 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220876 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220879 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220881 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220884 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220886 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220889 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220892 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220894 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220897 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220900 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220903 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220905 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220908 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220910 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220913 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220915 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220918 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.222790 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220921 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220923 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220925 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220928 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220931 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220934 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220936 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220939 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220941 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220944 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220946 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220949 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220951 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220955 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220957 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220960 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220962 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220965 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220967 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220971 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.223318 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220975 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220980 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220984 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220988 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220992 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.220996 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221001 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221003 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221006 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221009 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221011 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221014 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221016 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221019 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221022 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221024 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221027 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221030 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221040 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221044 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.223846 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221047 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221049 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221052 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221055 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221057 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221060 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221063 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221065 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221069 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221073 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221076 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221079 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221082 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221085 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221087 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221090 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221092 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221095 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221097 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.224349 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221100 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221103 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221105 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221108 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221111 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221113 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221117 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221121 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221123 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221126 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221129 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221134 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221142 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221145 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221148 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221150 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221648 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221654 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221657 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.224827 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221660 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221663 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221665 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221668 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221671 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221673 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221676 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221678 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221681 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221684 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221687 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221690 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221692 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221707 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221710 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221713 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221716 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221718 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221721 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.225285 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221724 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221726 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221729 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221732 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221734 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221739 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221742 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221745 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221748 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221750 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221753 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221755 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221758 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221761 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221763 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221766 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221768 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221771 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221773 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221776 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.225775 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221778 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221781 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221783 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221786 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221789 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221793 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221796 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221800 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221804 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221806 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221809 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221812 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221814 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221817 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221819 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221822 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221825 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221829 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221832 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.226260 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221836 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221838 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221841 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221843 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221846 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221848 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221850 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221853 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221856 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221859 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221861 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221864 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221866 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221869 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221871 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221874 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221876 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221879 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221881 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221884 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.226739 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221887 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221889 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221892 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221895 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221897 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221900 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221902 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221905 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221907 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221910 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221914 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221916 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221919 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221928 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221931 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221933 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221938 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221942 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221945 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.227331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221949 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221952 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221954 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221957 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221959 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.221962 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222056 4524 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222066 4524 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222081 4524 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222088 4524 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222095 4524 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222098 4524 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222102 4524 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222107 4524 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222110 4524 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222113 4524 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222116 4524 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222120 4524 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222123 4524 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222126 4524 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222129 4524 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222132 4524 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222135 4524 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:41:45.227817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222140 4524 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222145 4524 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222148 4524 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222151 4524 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222158 4524 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222162 4524 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222166 4524 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222169 4524 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222172 4524 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222176 4524 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222179 4524 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222182 4524 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222185 4524 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222189 4524 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222191 4524 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222196 4524 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222199 4524 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222202 4524 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222205 4524 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222208 4524 flags.go:64] FLAG: --enable-server="true" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222211 4524 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222215 4524 flags.go:64] FLAG: --event-burst="100" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222221 4524 flags.go:64] FLAG: --event-qps="50" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222224 4524 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:41:45.228401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222227 4524 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222230 4524 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222235 4524 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222237 4524 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222241 4524 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222244 4524 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222247 4524 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222250 4524 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222255 4524 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222258 4524 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222261 4524 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222264 4524 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222267 4524 flags.go:64] FLAG: --feature-gates="" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222271 4524 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222275 4524 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222278 4524 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222282 4524 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222287 4524 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222291 4524 flags.go:64] FLAG: --help="false" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222294 4524 flags.go:64] FLAG: --hostname-override="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222298 4524 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222301 4524 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222304 4524 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:41:45.229041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222307 4524 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222310 4524 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222313 4524 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222316 4524 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222319 4524 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222322 4524 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222325 4524 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222328 4524 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222331 4524 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222334 4524 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222336 4524 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222340 4524 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222342 4524 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222345 4524 flags.go:64] FLAG: --lock-file="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222348 4524 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222351 4524 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222354 4524 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222361 4524 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222364 4524 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222367 4524 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222370 4524 flags.go:64] FLAG: --logging-format="text" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222373 4524 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222376 4524 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222380 4524 flags.go:64] FLAG: --manifest-url="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222384 4524 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:41:45.229688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222389 4524 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222393 4524 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222397 4524 flags.go:64] FLAG: --max-pods="110" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222400 4524 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222403 4524 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222406 4524 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222409 4524 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222412 4524 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222415 4524 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222418 4524 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222424 4524 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222427 4524 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222430 4524 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222433 4524 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222436 4524 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222439 4524 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222443 4524 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222446 4524 flags.go:64] FLAG: --port="10250" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222448 4524 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222451 4524 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01e47e2e893ca1655" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222455 4524 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222458 4524 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222461 4524 flags.go:64] FLAG: --register-node="true" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222463 4524 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222470 4524 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222473 4524 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:41:45.230485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222475 4524 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222478 4524 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222482 4524 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222485 4524 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222489 4524 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222492 4524 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222495 4524 flags.go:64] FLAG: --runonce="false" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222499 4524 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222502 4524 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222505 4524 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222508 4524 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222511 4524 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222514 4524 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222517 4524 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222520 4524 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222523 4524 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222526 4524 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222529 4524 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222532 4524 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222535 4524 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222538 4524 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222541 4524 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222546 4524 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222549 4524 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222552 4524 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:41:45.231366 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222558 4524 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222560 4524 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222563 4524 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222566 4524 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222569 4524 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222574 4524 flags.go:64] FLAG: --v="2" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222579 4524 flags.go:64] FLAG: --version="false" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222583 4524 flags.go:64] FLAG: --vmodule="" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222587 4524 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.222590 4524 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222737 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222742 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222745 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222750 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222754 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222758 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222762 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222766 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222769 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222772 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222776 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.232031 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222778 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222781 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222783 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222786 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222788 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222792 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222795 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222797 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222800 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222803 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222806 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222809 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222811 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222814 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222816 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222819 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222823 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222825 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222829 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.232668 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222833 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222836 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222839 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222842 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222844 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222847 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222851 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222854 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222857 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222859 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222862 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222865 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222867 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222870 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222873 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222876 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222878 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222881 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222884 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.233203 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222886 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222889 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222891 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222894 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222897 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222899 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222902 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222904 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222907 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222910 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222912 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222915 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222917 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222920 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222922 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222925 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222927 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222930 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222932 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222936 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.233994 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222939 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222942 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222945 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222948 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222950 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222953 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222955 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222958 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222961 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222963 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222966 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222969 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222971 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222974 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222977 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222979 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222982 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222984 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222987 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222990 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.234958 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222992 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222995 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.222997 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223000 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223002 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223005 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223007 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223010 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223013 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223015 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223018 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223021 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223024 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223027 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223030 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223032 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.235522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223035 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.223060 4524 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223095 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223100 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223104 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223106 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223109 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223112 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223114 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223117 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223120 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223122 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223125 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223127 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223130 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223133 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.235943 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223135 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223138 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223140 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223143 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223146 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223148 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223151 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223153 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223156 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223159 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223161 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223164 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223168 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223172 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223175 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223178 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223180 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223182 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223185 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223188 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.236375 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223190 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223193 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223195 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223198 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223200 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223203 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223206 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223208 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223211 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223213 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223216 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223219 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223221 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223224 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223226 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223229 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223232 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223234 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223237 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223240 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.236874 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223242 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223246 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223250 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223253 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223257 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223261 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223263 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223266 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223269 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223272 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223275 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223277 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223280 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223282 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223285 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223288 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223290 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223294 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223297 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.237353 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223300 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223302 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223305 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223307 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223310 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223314 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223318 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223321 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223324 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223327 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223330 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223333 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223336 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223338 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223341 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223344 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223346 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223349 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.237887 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223353 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223356 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223359 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223362 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223365 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223367 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223370 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223372 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223375 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223377 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223380 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223383 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223386 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223389 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.223391 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.226841 4524 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.226858 4524 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.226876 4524 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.226880 4524 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:41:45.238470 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226933 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226939 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226942 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226945 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226948 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226951 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226954 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226957 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226960 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226962 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226965 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226968 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226970 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226973 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226975 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226978 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226981 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226983 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226986 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226988 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.239331 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226991 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226994 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226996 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.226999 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227002 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227005 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227009 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227013 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227016 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227020 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227023 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227026 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227028 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227031 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227034 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227036 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227039 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227042 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227045 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.239845 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227047 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227050 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227052 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227054 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227057 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227060 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227062 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227065 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227068 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227071 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227073 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227076 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227079 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227081 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227084 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227087 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227090 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227093 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227096 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.240303 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227098 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227101 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227104 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227107 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227110 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227113 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227115 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227118 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227120 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227123 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227126 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227129 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227131 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227134 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227137 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227139 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227142 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227145 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227148 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227150 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.240778 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227153 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227155 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227158 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227160 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227163 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227165 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227168 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227171 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227173 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227176 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227179 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227181 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227184 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227186 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227189 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227191 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227194 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227196 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227199 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227202 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.241316 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227204 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227208 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227212 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227215 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227218 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227220 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227223 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227225 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.227252 4524 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227391 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227396 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227399 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227402 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227404 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227407 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227410 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.242040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227412 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227415 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227417 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227420 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227422 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227425 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227427 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227430 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227433 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227435 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227438 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227440 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227443 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227446 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227449 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227451 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227454 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227456 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227459 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.242560 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227461 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227464 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227467 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227469 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227472 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227475 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227477 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227480 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227483 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227485 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227488 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227490 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227493 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227495 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227498 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227500 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227503 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227505 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227508 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227510 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.243104 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227513 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227515 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227518 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227521 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227523 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227526 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227528 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227531 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227534 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227537 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227539 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227542 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227547 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227550 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227552 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227555 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227558 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227560 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227563 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227565 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.243657 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227568 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227570 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227573 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227577 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227581 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227583 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227586 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227588 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227591 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227593 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227598 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227601 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227603 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227606 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227608 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227611 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227614 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227616 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227619 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227622 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.244229 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227624 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227627 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227630 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227633 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227636 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227639 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227641 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227644 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227647 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227649 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227652 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227654 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227657 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227659 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227663 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227667 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227670 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227673 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227676 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.244840 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227678 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.227713 4524 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227744 4524 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227748 4524 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227753 4524 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227756 4524 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227759 4524 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227762 4524 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227764 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227767 4524 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227769 4524 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227772 4524 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227774 4524 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227777 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227780 4524 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227783 4524 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:41:45.245515 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227786 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227790 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227793 4524 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227795 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227798 4524 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227801 4524 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227803 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227806 4524 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227808 4524 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227811 4524 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227814 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227816 4524 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227819 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227822 4524 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227825 4524 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227827 4524 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227831 4524 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227833 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227836 4524 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227839 4524 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:41:45.245921 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227843 4524 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227845 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227848 4524 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227851 4524 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227855 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227858 4524 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227861 4524 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227864 4524 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227866 4524 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227869 4524 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227871 4524 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227874 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227877 4524 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227881 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227884 4524 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227886 4524 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227889 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227892 4524 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227894 4524 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227897 4524 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:41:45.246473 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227899 4524 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227902 4524 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227904 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227907 4524 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227910 4524 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227912 4524 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227915 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227917 4524 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227920 4524 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227923 4524 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227925 4524 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227928 4524 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227931 4524 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227935 4524 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227939 4524 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227942 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227945 4524 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227947 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227950 4524 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:41:45.247368 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227953 4524 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227955 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227958 4524 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227961 4524 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227963 4524 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227966 4524 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227970 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227973 4524 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227975 4524 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227978 4524 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227980 4524 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227983 4524 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227986 4524 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227988 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227991 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227993 4524 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227996 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.227999 4524 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228001 4524 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:41:45.248141 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228004 4524 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228006 4524 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228009 4524 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228011 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228014 4524 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228016 4524 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228021 4524 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228023 4524 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228026 4524 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228028 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228031 4524 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228034 4524 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228036 4524 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:45.228039 4524 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.228091 4524 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229029 4524 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229084 4524 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229257 4524 server.go:1014] "Starting client certificate rotation" Oct 30 19:41:45.248858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229392 4524 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229440 4524 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:03 +0000 UTC" deadline="2028-05-11 21:09:52.331196802 +0000 UTC" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229467 4524 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13417h28m7.101731396s" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.229668 4524 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.231728 4524 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.233019 4524 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.234836 4524 log.go:25] "Validated CRI v1 image API" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.235581 4524 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:41:45.249297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.236990 4524 fs.go:135] Filesystem UUIDs: map[60386cfd-a6b2-4c7d-8596-23556d32c656:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b7fe18c1-03e7-4ef2-90fe-2e2416b6a3a0:/dev/nvme0n1p3] Oct 30 19:41:45.249509 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.237010 4524 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24b7a886-5347-4494-a757-db73e0177366/volumes/kubernetes.io~projected/kube-api-access-6qblk:{mountpoint:/var/lib/kubelet/pods/24b7a886-5347-4494-a757-db73e0177366/volumes/kubernetes.io~projected/kube-api-access-6qblk major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~projected/kube-api-access-x8rqg:{mountpoint:/var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~projected/kube-api-access-x8rqg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~projected/kube-api-access-wzqr8:{mountpoint:/var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~projected/kube-api-access-wzqr8 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~projected/kube-api-access-wchwj:{mountpoint:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~projected/kube-api-access-wchwj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bf076c5-f276-4191-9240-8968d8b20978/volumes/kubernetes.io~projected/kube-api-access-66fvf:{mountpoint:/var/lib/kubelet/pods/3bf076c5-f276-4191-9240-8968d8b20978/volumes/kubernetes.io~projected/kube-api-access-66fvf major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327/volumes/kubernetes.io~projected/kube-api-access-vt8q8:{mountpoint:/var/lib/kubelet/pods/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327/volumes/kubernetes.io~projected/kube-api-access-vt8q8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e60683e-1ea5-4951-aae1-c76f68ef297a/volumes/kubernetes.io~projected/kube-api-access-xsz49:{mountpoint:/var/lib/kubelet/pods/6e60683e-1ea5-4951-aae1-c76f68ef297a/volumes/kubernetes.io~projected/kube-api-access-xsz49 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eca3fa5-f093-4f9a-a7e7-d19fff01690f/volumes/kubernetes.io~projected/kube-api-access-n8zgz:{mountpoint:/var/lib/kubelet/pods/6eca3fa5-f093-4f9a-a7e7-d19fff01690f/volumes/kubernetes.io~projected/kube-api-access-n8zgz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80ef5a23-6398-4f49-b7f3-09f18b3473ca/volumes/kubernetes.io~projected/kube-api-access-629cq:{mountpoint:/var/lib/kubelet/pods/80ef5a23-6398-4f49-b7f3-09f18b3473ca/volumes/kubernetes.io~projected/kube-api-access-629cq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae6f73a8-cc0c-4c3a-b771-90434a04fc32/volumes/kubernetes.io~projected/kube-api-access-m9g76:{mountpoint:/var/lib/kubelet/pods/ae6f73a8-cc0c-4c3a-b771-90434a04fc32/volumes/kubernetes.io~projected/kube-api-access-m9g76 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4dc42eb-2297-455c-a982-d1f42131323c/volumes/kubernetes.io~projected/kube-api-access-sgqqc:{mountpoint:/var/lib/kubelet/pods/b4dc42eb-2297-455c-a982-d1f42131323c/volumes/kubernetes.io~projected/kube-api-access-sgqqc major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdfc0cb5-a33f-488c-974e-75c43659c08a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cdfc0cb5-a33f-488c-974e-75c43659c08a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1d03ece-45f6-4355-8414-729003908b01/volumes/kubernetes.io~projected/kube-api-access-wbtf7:{mountpoint:/var/lib/kubelet/pods/d1d03ece-45f6-4355-8414-729003908b01/volumes/kubernetes.io~projected/kube-api-access-wbtf7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3626c0-00b5-4039-a06c-9537f72283bc/volumes/kubernetes.io~projected/kube-api-access-jb6fk:{mountpoint:/var/lib/kubelet/pods/dd3626c0-00b5-4039-a06c-9537f72283bc/volumes/kubernetes.io~projected/kube-api-access-jb6fk major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~projected/kube-api-access-fllq9:{mountpoint:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~projected/kube-api-access-fllq9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1124e17-db0a-42d9-8be6-82db4177cf24/volumes/kubernetes.io~projected/kube-api-access-wjf69:{mountpoint:/var/lib/kubelet/pods/e1124e17-db0a-42d9-8be6-82db4177cf24/volumes/kubernetes.io~projected/kube-api-access-wjf69 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e14438f5-ecd7-4a84-9bb4-7648b87bd075/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e14438f5-ecd7-4a84-9bb4-7648b87bd075/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efe11c10-4c93-48fc-a213-0faaf29b5d11/volumes/kubernetes.io~projected/kube-api-access-5mg7d:{mountpoint:/var/lib/kubelet/pods/efe11c10-4c93-48fc-a213-0faaf29b5d11/volumes/kubernetes.io~projected/kube-api-access-5mg7d major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1ae965cc92ba99074192f350fab2cdfddcc9eacabf23b187811e088fadba58a4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a3b3d5849186bdf407d6649ffcb575e4cd4beea3dc3430649ce9734b98583dbb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/88c870f73f1d1ea9c609dedd19d518802bf74630f8863e1ca0be296666f61150/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/68c465d7b751e7a0955df83fb936499b3df3b6e9fd2ac0c56c9b2312e09f92a9/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/95d376db77917806eebc77acb6703b4ea754017176432f5ea847f8b5cb6b4a41/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cdd8a50239edaf1760522e30e6ed6268f8d7338d8b1b9e10ef3fa01972c5122d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f6aeb96a3722f7514c20288e66209391a152818df7af260ce5832d9205bbf973/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8c0db51d796855e01973059ba85bccf6f6df74ae9539f367dfafc56905e2fa1a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e7c79a6ee4fcb196dbbde49eb9e4819681511ee7683c0e6d2b54939dbc31685b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/207d77ed96921f2a3ed9a5a870f6a782c9a747a3fbf146e6c328d0a8959043b9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b7aa8f89247c607aa5f4aa6f2487cbae27dfda4da0926b5aaa76ebf4e8606055/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/d34c2e618fdc5069abd461e7be736bf8d19d4b025a1aca8b90fe349b560ae0ed/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/798a4b8ab3162996162aa313a80f0ce4e5bd572a2476aa95622d3a5e45cd61a9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/32517aba0ec243597fdebe1898e8283e4a18952067353f0162e850d02c2e8ee4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/88be4b3401c50c03ad20408e04ed0ae74f23f4dfbac72f1b093bb5fc78f0cd1f/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9df0d759ec1ae69846a30b4b81aedb4f037c1f3e9b9353cd0114e42af96de619/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/113ac282a4d57562812d7a4d5875b4b1d0aa8962acdc6297d80f7de7b4ff9418/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f85bf55b3faa1a64263969fcbb05e248cff3ba238647dca6d5af6474f6416099/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dcb4513674f6b40445699f26cf458d5635950684b4053dac8ed4de364af45954/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1c8dc8e7cf83c67f46a56171c19d878d9d19e321d76d82ad0aeb55a97332abdc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/52d50aee88d473fe75838a7dd2d0f3c44e8cd44becade8b2111922951c7e3744/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/e3f4b6b0feb092568fb5da585e22bef6502c56e3787af9835f2411d0ce5286c8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/67c8edf1801994b274c18c970493a662dc41bcbeffbe399a416d77fa355ea674/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/66a5aaaf9812c2053528c4005c99ae1b2341599404d01b5995e7c3c9b9280e9d/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/d3812fa7b8b39ed14b49c622112c783a78351690c4cb781d2172ad6b91dcc313/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/8034f1e52a2d679f7dedfeeee21fbef17b3956d423eb9cba6c1ebd0be0e0a350/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/78ff21998f3cb71e35ea193bc8ad41a2048da9eae2621083238eda9dd98e7bd6/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/1d20332c58c780ef66961fdc4211d366fe9c451909fd69674523757e13f01284/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/2d69863c95f3c0464c51afc2b64ded5c0a0af5529497361062f56280791fd984/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/3c0818f6a5ade2aa470e05908a6443aede63bc82a6ee08c0004ea8af963c2406/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/ba06d1baee98a9ee73b26aaa7d826397d14c6be5d39e70b493d26c4da9291195/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5d1489c37a036e59100424c097b5ff7346503479b605bd01300f76803155b374/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5ca9ca8ae6b04d319852e34cfcd88099117d46b677bb79253f1ad7eccf394a37/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cd9cc33195ed2a89cacbfad40fff0824cc3099c15d6b059bad67bde4e046ee34/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/54094a61e0eadc7ee9af9901e09f97e48cea991aec29fefa3a62de0aa5728693/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e5b4c1071500de4d0158bfbd64252585a02f9ada4bbd912c3ece7e009c9fd49c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/39fd4f101ce5b241bed25e4218d72254e3eef00664df5ff2e2187e4d05a627c1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/879a7259f71d3269db17576f388b29a2b56c021426d6554dbbfeef5d6f3ae64e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ba551611637d2041f941d9eb575f70d8da42d13e05b7aa1d809f41c07e424ae3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f53b00c3b334ab8b3db3d09bac845e3fe1b797cb6581c961651c700cb7a7beaf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9c33942f7e44af38bad65c456c616fdb21aa8a38caa7387739636ffec6a89174/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.242900 4524 manager.go:217] Machine: {Timestamp:2026-10-30 19:41:45.242447127 +0000 UTC m=+0.086118754 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097691 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2ecd5319b4579e5b28e173fdc453 SystemUUID:ec2d2ecd-5319-b457-9e5b-28e173fdc453 BootID:95106406-f6a3-46a0-9648-d8e64be07aac Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1d03ece-45f6-4355-8414-729003908b01/volumes/kubernetes.io~projected/kube-api-access-wbtf7 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cad7ece307507f285f58c20b2172e49b8ccb5ac447f722c9d4fd8c9d3f96c3fd/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~projected/kube-api-access-wzqr8 DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80ef5a23-6398-4f49-b7f3-09f18b3473ca/volumes/kubernetes.io~projected/kube-api-access-629cq DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1124e17-db0a-42d9-8be6-82db4177cf24/volumes/kubernetes.io~projected/kube-api-access-wjf69 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f31f1a101a417602b84762c348644a8b70da53fb1c6a12104eec0178a46bd1ce/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39e81042eaef0614239949f7d515be4eac1c367d40000a9cf7783232e3106327/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/260bbf80-e9c9-4264-8983-7bf37d39a563/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~projected/kube-api-access-fllq9 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae6f73a8-cc0c-4c3a-b771-90434a04fc32/volumes/kubernetes.io~projected/kube-api-access-m9g76 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d53fad80340ab23901b9b829e4a49889a46aa9eab93cc5f068c6cee96c22d8c2/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29f3090e6e81fcaff93e0bbd2eccebb57e982f50022fc25520b771f758abbd77/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e14438f5-ecd7-4a84-9bb4-7648b87bd075/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3626c0-00b5-4039-a06c-9537f72283bc/volumes/kubernetes.io~projected/kube-api-access-jb6fk DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b4dc42eb-2297-455c-a982-d1f42131323c/volumes/kubernetes.io~projected/kube-api-access-sgqqc DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/955a0198309597cae876d610d812586167ec1f304bbf3e47d0578587bcf94e70/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a62f409e84040c652b4445bded9ca3a48e116985e18798007975b9da151f3664/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62efc0449d30050b942938ed2d9612796c5d28fad35c07e0ce1f6dfa5c2b2f87/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6eca3fa5-f093-4f9a-a7e7-d19fff01690f/volumes/kubernetes.io~projected/kube-api-access-n8zgz DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327/volumes/kubernetes.io~projected/kube-api-access-vt8q8 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24f786ff-dc91-4035-96cc-2e840a7fcae5/volumes/kubernetes.io~projected/kube-api-access-x8rqg DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bf076c5-f276-4191-9240-8968d8b20978/volumes/kubernetes.io~projected/kube-api-access-66fvf DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bccc40de5d07670b1f9bd12ba039f5e7b4ba4ba627ed7ba95320226a0d67910/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e60683e-1ea5-4951-aae1-c76f68ef297a/volumes/kubernetes.io~projected/kube-api-access-xsz49 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad31613f561e9cb38ca89a054dbd465d9bb073b3d0bdc963c6f111e29f1527aa/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdfc0cb5-a33f-488c-974e-75c43659c08a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/efe11c10-4c93-48fc-a213-0faaf29b5d11/volumes/kubernetes.io~projected/kube-api-access-5mg7d DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7/volumes/kubernetes.io~projected/kube-api-access-wchwj DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0fea14805912979ad5b3bcefad3736f300f8ad871dfdafa9bc42a7f3be8aa35/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66bc8a49d93c0f0d409718349f29f6632dc4695a1525724a81eb9f2292b1ac91/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0525652-ce3f-4898-9be8-066de957058e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90c9e93129a0e7c861332ae01eb85d266bbd30fafa8f592a7d659e8d4f1f8c74/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24b7a886-5347-4494-a757-db73e0177366/volumes/kubernetes.io~projected/kube-api-access-6qblk DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:29f3090e6e81fca MacAddress:ca:50:b6:d1:f5:ee Speed:10000 Mtu:8901} {Name:2c631ef2717ff78 MacAddress:ea:e7:c2:0d:e5:94 Speed:10000 Mtu:8901} {Name:39e81042eaef061 MacAddress:de:02:6f:4b:eb:e6 Speed:10000 Mtu:8901} {Name:62efc0449d30050 MacAddress:92:9d:bd:ce:24:e0 Speed:10000 Mtu:8901} {Name:ad31613f561e9cb MacAddress:ce:61:13:f2:5f:08 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:31:02:49:f3:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:31:02:4d:c9:49 Speed:0 Mtu:8901} {Name:d53fad80340ab23 MacAddress:c2:26:a3:a0:e0:1f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:31:02:49:f3:09 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fa:78:df:9f:e0:ba Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:c4:37:db:e5:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243138 4524 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243176 4524 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243438 4524 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243459 4524 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-41.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243568 4524 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243576 4524 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243585 4524 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243616 4524 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243714 4524 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243858 4524 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.243973 4524 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244005 4524 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244015 4524 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244025 4524 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244033 4524 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244040 4524 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244734 4524 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.244748 4524 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:41:45.249644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.245194 4524 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:41:45.255340 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255326 4524 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:41:45.255384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255342 4524 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:41:45.255428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255419 4524 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:41:45.255601 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255590 4524 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:41:45.255632 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255619 4524 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:41:45.255728 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255719 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255734 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255741 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255747 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255752 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255758 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255763 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:41:45.255769 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255769 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255775 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255786 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255794 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255805 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255831 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:41:45.256054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.255837 4524 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:41:45.259016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.259004 4524 server.go:1263] "Started kubelet" Oct 30 19:41:45.259384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.259082 4524 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:41:45.259384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.259099 4524 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:41:45.259384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.259251 4524 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:41:45.259599 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.259527 4524 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:41:45.259548 ip-10-0-130-41 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:41:45.260371 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.260355 4524 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262093 4524 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262139 4524 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262153 4524 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262168 4524 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.262204 4524 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262195 4524 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:05 +0000 UTC" deadline="2028-06-25 20:57:42.262202153 +0000 UTC" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262248 4524 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14497h15m56.999958191s" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.262267 4524 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264524 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae6f73a8-cc0c-4c3a-b771-90434a04fc32" volumeName="kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264590 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4dc42eb-2297-455c-a982-d1f42131323c" volumeName="kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264617 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0525652-ce3f-4898-9be8-066de957058e" volumeName="kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264625 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1124e17-db0a-42d9-8be6-82db4177cf24" volumeName="kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264644 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe11c10-4c93-48fc-a213-0faaf29b5d11" volumeName="kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264653 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24f786ff-dc91-4035-96cc-2e840a7fcae5" volumeName="kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264659 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260bbf80-e9c9-4264-8983-7bf37d39a563" volumeName="kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264673 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1efff8-ab0b-4bde-841c-cce83e4ad6c7" volumeName="kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264685 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e5c9fe7-ca03-478a-85ad-1a6e4afb4327" volumeName="kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264726 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80ef5a23-6398-4f49-b7f3-09f18b3473ca" volumeName="kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264744 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0525652-ce3f-4898-9be8-066de957058e" volumeName="kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264757 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1124e17-db0a-42d9-8be6-82db4177cf24" volumeName="kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264773 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe11c10-4c93-48fc-a213-0faaf29b5d11" volumeName="kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264801 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260bbf80-e9c9-4264-8983-7bf37d39a563" volumeName="kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264812 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf076c5-f276-4191-9240-8968d8b20978" volumeName="kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264824 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e60683e-1ea5-4951-aae1-c76f68ef297a" volumeName="kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264846 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae6f73a8-cc0c-4c3a-b771-90434a04fc32" volumeName="kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264882 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0525652-ce3f-4898-9be8-066de957058e" volumeName="kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264888 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe11c10-4c93-48fc-a213-0faaf29b5d11" volumeName="kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264903 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1efff8-ab0b-4bde-841c-cce83e4ad6c7" volumeName="kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264915 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d03ece-45f6-4355-8414-729003908b01" volumeName="kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264923 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3626c0-00b5-4039-a06c-9537f72283bc" volumeName="kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264944 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe11c10-4c93-48fc-a213-0faaf29b5d11" volumeName="kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264955 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf076c5-f276-4191-9240-8968d8b20978" volumeName="kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.264970 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e60683e-1ea5-4951-aae1-c76f68ef297a" volumeName="kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265000 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eca3fa5-f093-4f9a-a7e7-d19fff01690f" volumeName="kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265013 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80ef5a23-6398-4f49-b7f3-09f18b3473ca" volumeName="kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265056 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0525652-ce3f-4898-9be8-066de957058e" volumeName="kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265097 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdfc0cb5-a33f-488c-974e-75c43659c08a" volumeName="kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265113 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b7a886-5347-4494-a757-db73e0177366" volumeName="kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265155 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24f786ff-dc91-4035-96cc-2e840a7fcae5" volumeName="kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265175 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4dc42eb-2297-455c-a982-d1f42131323c" volumeName="kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265186 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0525652-ce3f-4898-9be8-066de957058e" volumeName="kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265221 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1efff8-ab0b-4bde-841c-cce83e4ad6c7" volumeName="kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265260 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260bbf80-e9c9-4264-8983-7bf37d39a563" volumeName="kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265273 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae6f73a8-cc0c-4c3a-b771-90434a04fc32" volumeName="kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265283 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d03ece-45f6-4355-8414-729003908b01" volumeName="kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265303 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1124e17-db0a-42d9-8be6-82db4177cf24" volumeName="kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265311 4524 factory.go:55] Registering systemd factory Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265335 4524 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265314 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e14438f5-ecd7-4a84-9bb4-7648b87bd075" volumeName="kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265425 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e14438f5-ecd7-4a84-9bb4-7648b87bd075" volumeName="kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265477 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf076c5-f276-4191-9240-8968d8b20978" volumeName="kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265498 4524 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf076c5-f276-4191-9240-8968d8b20978" volumeName="kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265511 4524 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265521 4524 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265581 4524 factory.go:153] Registering CRI-O factory Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265599 4524 factory.go:223] Registration of the crio container factory successfully Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265674 4524 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265724 4524 factory.go:103] Registering Raw factory Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.265742 4524 manager.go:1199] Started watching for new ooms in manager Oct 30 19:41:45.267615 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.266314 4524 manager.go:319] Starting recovery of all containers Oct 30 19:41:45.290345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.290286 4524 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:41:45.291944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.291920 4524 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:41:45.291944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.291939 4524 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:41:45.292075 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.291970 4524 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:41:45.292075 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.292003 4524 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:41:45.304409 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.304385 4524 generic.go:356] "Generic (PLEG): container finished" podID="4a99c2ea9ac9c2ae64a2a379a47ab1fe" containerID="6cb11a7fa7c0552afff11f79f60f2cce9e051ec2fc3c8d3a5fcce090d1a302a2" exitCode=0 Oct 30 19:41:45.327906 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327881 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="5b989f07f65246eafbf588ca197f3bd99f912e92a22a466dc912a44dd2b57bd7" exitCode=0 Oct 30 19:41:45.327906 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327905 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="3c9f162db630f315c41ef23913c62c42a2d859722f5622c418c77fbd9e186753" exitCode=0 Oct 30 19:41:45.328071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327917 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="efc23853816df6694989f2c602cf68cb7d6bccab2d5bf8a747d0d3a060362f34" exitCode=0 Oct 30 19:41:45.328071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327927 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="333faff876a057bbf4cdb545115e821a3d94f525f6c3e16b184381ef00d48739" exitCode=0 Oct 30 19:41:45.328071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327936 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="106d4ee8c6a60d552adf57bd709577c35f701175fee311aed9faa5367303dbcd" exitCode=0 Oct 30 19:41:45.328071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.327943 4524 generic.go:356] "Generic (PLEG): container finished" podID="efe11c10-4c93-48fc-a213-0faaf29b5d11" containerID="a0988406e69cca93c3b9f1cb8b2196c6e36d5ca421c77e0678f12b0d7ef33851" exitCode=0 Oct 30 19:41:45.338597 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.338583 4524 manager.go:324] Recovery completed Oct 30 19:41:45.352534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.352513 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.353584 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.353570 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.353644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.353598 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.353644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.353609 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.354900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.354860 4524 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:41:45.354900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.354872 4524 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:41:45.354900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.354877 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.354900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.354888 4524 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.354999 4524 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355012 4524 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355030 4524 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355036 4524 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355044 4524 policy_none.go:50] "Start" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355053 4524 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:41:45.355073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355062 4524 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:41:45.355349 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355208 4524 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:41:45.355349 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355217 4524 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:41:45.355349 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355223 4524 policy_none.go:44] "Start" Oct 30 19:41:45.355741 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355725 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.355804 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355756 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.355804 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.355771 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.359202 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.359181 4524 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:41:45.361735 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.361718 4524 manager.go:341] "Starting Device Plugin manager" Oct 30 19:41:45.361813 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.361747 4524 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:41:45.361813 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.361765 4524 server.go:87] "Starting device plugin registration server" Oct 30 19:41:45.362061 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362048 4524 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:41:45.362120 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362062 4524 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:41:45.362198 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362178 4524 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:41:45.362352 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362309 4524 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:41:45.362352 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362352 4524 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:41:45.362495 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362359 4524 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:41:45.362495 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.362409 4524 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:41:45.362342122Z","UUID":"99eaf9b4-1088-4afa-ba76-ed1fe42dc9d8","Handler":null,"Name":"","Endpoint":""} Oct 30 19:41:45.363550 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.363531 4524 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:41:45.363625 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.363575 4524 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:41:45.363625 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.363588 4524 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" Oct 30 19:41:45.363625 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.363598 4524 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:41:45.392762 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.392729 4524 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal"] Oct 30 19:41:45.392845 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.392835 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.393719 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.393693 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.393788 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.393728 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.393788 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.393738 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.393963 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.393951 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.394098 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394083 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.394759 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394742 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.394848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394774 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.394848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394748 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.394848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394791 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.394848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394801 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.394848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.394816 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.394987 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.394976 4524 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.395068 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395044 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerStarted","Data":"70a923ff705cd5d0f3b54112c28e1001f83fef1dfb96917d32afcf359bcbfc6f"} Oct 30 19:41:45.395099 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395078 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerDied","Data":"6cb11a7fa7c0552afff11f79f60f2cce9e051ec2fc3c8d3a5fcce090d1a302a2"} Oct 30 19:41:45.395126 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395100 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"4a99c2ea9ac9c2ae64a2a379a47ab1fe","Type":"ContainerStarted","Data":"1dc3813f3cb84636c0ec7203aeeaea1083a184cc78f8d72716b43d9350f0839a"} Oct 30 19:41:45.395126 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395119 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"baf763e8cc3376af1056e0b1dfc87c5b","Type":"ContainerStarted","Data":"7e1005395cdaa3c37812e9163f78af1e53c9b7ff1d4e441a441f9b1b80b3d271"} Oct 30 19:41:45.395178 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395135 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" event={"ID":"baf763e8cc3376af1056e0b1dfc87c5b","Type":"ContainerStarted","Data":"f759e1c25760a7e40f068c2ebf0a5d3490e8b13eb4f701f2f94594f8ad278c68"} Oct 30 19:41:45.395178 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395152 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.395984 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395967 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.396054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.395992 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.396054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.396002 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.396134 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.396113 4524 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-41.us-west-1.compute.internal\" not found" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.462454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.462435 4524 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:41:45.463248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.463234 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:41:45.463308 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.463258 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:41:45.463308 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.463268 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:41:45.463308 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.463292 4524 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.472704 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.472682 4524 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:45.474540 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.474526 4524 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:41:45.522515 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.522494 4524 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:45.524651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.524632 4524 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:45.524744 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.524726 4524 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:41:45.562683 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.562664 4524 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.574664 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.574642 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.574764 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.574673 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.574764 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.574717 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.577662 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.577641 4524 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.577746 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.577690 4524 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.578137 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.578125 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeNotReady" Oct 30 19:41:45.578184 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.578149 4524 setters.go:546] "Node became not ready" node="ip-10-0-130-41.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:41:45Z","lastTransitionTime":"2026-10-30T19:41:45Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 19:41:45.625460 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.625411 4524 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-41.us-west-1.compute.internal" is owned by different node (oldNodeID="19bfb0be-fe20-47c2-91be-f734ce72909b", newNodeID=""), cleaning up... Oct 30 19:41:45.625651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.625635 4524 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:45.625725 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.625685 4524 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.625725 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.625718 4524 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675664 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675643 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675666 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675682 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675732 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675759 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/baf763e8cc3376af1056e0b1dfc87c5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"baf763e8cc3376af1056e0b1dfc87c5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.675791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.675771 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a99c2ea9ac9c2ae64a2a379a47ab1fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" (UID: \"4a99c2ea9ac9c2ae64a2a379a47ab1fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.684461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.684439 4524 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:45.684542 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:45.684475 4524 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:45.850212 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.850193 4524 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-41.us-west-1.compute.internal" event="NodeReady" Oct 30 19:41:45.850294 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:45.850261 4524 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:41:46.244934 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.244909 4524 apiserver.go:52] "Watching apiserver" Oct 30 19:41:46.334470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.334411 4524 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:46.334575 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.334544 4524 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:46.348134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.348114 4524 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:41:46.348590 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.348569 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh","openshift-dns/dns-default-78n2p","openshift-insights/insights-runtime-extractor-w9tj4","openshift-cluster-node-tuning-operator/tuned-nvffm","openshift-multus/multus-wbfff","openshift-multus/multus-additional-cni-plugins-lg47k","openshift-image-registry/node-ca-jk2pd","kube-system/konnectivity-agent-h2ktl","openshift-ingress-canary/ingress-canary-k8hkx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-4tzpr","openshift-network-diagnostics/network-check-target-tdkh8","kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal","openshift-dns/node-resolver-hkb4l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2","openshift-insights/periodic-gathering-h7q9z-4p4kw","openshift-network-operator/iptables-alerter-j2p4l","openshift-ovn-kubernetes/ovnkube-node-g8v6z","openshift-service-ca/service-ca-5fffc97bdf-g9npz","kube-system/global-pull-secret-syncer-jqblf"] Oct 30 19:41:46.349010 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.348993 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.356006 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.355985 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.356972 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.356956 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.363387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.363371 4524 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:41:46.379803 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.379781 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.379858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.379812 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.379902 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.379859 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:46.379953 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.379898 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.380001 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.379987 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380073 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380057 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.380152 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380114 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380207 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380155 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.380255 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380202 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8q8\" (UniqueName: \"kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.380255 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380226 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-tuned\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380228 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.380345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380294 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.380345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380315 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380332 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjf69\" (UniqueName: \"kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.380507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380348 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.380507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380372 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380463 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.380507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380490 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629cq\" (UniqueName: \"kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.380651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380530 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mg7d\" (UniqueName: \"kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.380651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380570 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zgz\" (UniqueName: \"kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.380651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380625 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchwj\" (UniqueName: \"kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380642 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380657 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380672 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380687 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380717 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380757 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380772 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.380858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380794 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380905 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380944 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.381054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.380976 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.381054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381001 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-tmp\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381034 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381186 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381062 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381186 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381110 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.381186 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381143 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.381306 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381218 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:46.381306 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381236 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381306 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381252 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381306 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381270 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381308 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381337 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381375 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.381414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381400 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qblk\" (UniqueName: \"kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk\") pod \"migrator-66df5d496b-tk7kh\" (UID: \"24b7a886-5347-4494-a757-db73e0177366\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:46.381513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381424 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381440 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.381569 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381508 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.381569 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381516 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ef5a23-6398-4f49-b7f3-09f18b3473ca-tmp-dir\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.381623 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381576 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.381623 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381595 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.381623 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381611 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rqg\" (UniqueName: \"kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:46.381718 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381628 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.381718 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381654 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsz49\" (UniqueName: \"kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.381718 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381677 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381802 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381735 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllq9\" (UniqueName: \"kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.381802 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381763 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.381802 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381787 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.381904 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381812 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.381904 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381838 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.381904 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381861 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.382050 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381897 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382050 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381937 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-tmp-dir\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.382050 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381947 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.382050 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.381992 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g76\" (UniqueName: \"kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.382168 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382093 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.382168 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382124 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzqr8\" (UniqueName: \"kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.382225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382176 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.382225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382194 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.382225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382211 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382357 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382234 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.382357 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382289 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.382357 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382327 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:46.382506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382359 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:46.382506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382385 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382412 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.382506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382439 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtf7\" (UniqueName: \"kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.382506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382479 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382506 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382536 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382557 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382572 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382579 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4dc42eb-2297-455c-a982-d1f42131323c-data-volume\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382608 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382637 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382659 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382681 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382712 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.382736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382727 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382793 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382818 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382833 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382851 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382875 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382902 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382917 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382963 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.382981 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383003 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383029 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.383062 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383045 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.383387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383069 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.383387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383087 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqqc\" (UniqueName: \"kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.383387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383103 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.383387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383119 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.383387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.383134 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.395478 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.395464 4524 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:46.395569 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.395503 4524 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:46.399990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.399977 4524 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:41:46.400053 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.400008 4524 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-41.us-west-1.compute.internal" Oct 30 19:41:46.405794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.405775 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:41:46.405794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.405786 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.405794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.405794 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.406016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406001 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:41:46.406070 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406028 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\"" Oct 30 19:41:46.406070 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406031 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\"" Oct 30 19:41:46.406070 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406064 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\"" Oct 30 19:41:46.406187 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406137 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\"" Oct 30 19:41:46.406187 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406141 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\"" Oct 30 19:41:46.406256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406185 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\"" Oct 30 19:41:46.406256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406185 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\"" Oct 30 19:41:46.406256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406186 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\"" Oct 30 19:41:46.406256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.406220 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:41:46.407330 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407308 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:41:46.407407 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407395 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.407519 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407506 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:41:46.407611 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407597 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.407665 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407608 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.407665 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407599 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:41:46.407775 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407663 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:41:46.407775 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407690 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:41:46.407775 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407735 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.407902 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.407878 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.408151 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.408137 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.408208 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.408163 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:41:46.410209 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.410194 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\"" Oct 30 19:41:46.410433 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.410418 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdfc0cb5-a33f-488c-974e-75c43659c08a-original-pull-secret\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.411530 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.411517 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e14438f5-ecd7-4a84-9bb4-7648b87bd075-konnectivity-ca\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:46.411601 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.411587 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-daemon-config\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.412716 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.412682 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e14438f5-ecd7-4a84-9bb4-7648b87bd075-agent-certs\") pod \"konnectivity-agent-h2ktl\" (UID: \"e14438f5-ecd7-4a84-9bb4-7648b87bd075\") " pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:41:46.412797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.412732 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60683e-1ea5-4951-aae1-c76f68ef297a-serviceca\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.412797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.412741 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.413850 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.413834 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1124e17-db0a-42d9-8be6-82db4177cf24-cni-binary-copy\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.413900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.413836 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.413900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.413874 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efe11c10-4c93-48fc-a213-0faaf29b5d11-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.464527 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.464498 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjf69\" (UniqueName: \"kubernetes.io/projected/e1124e17-db0a-42d9-8be6-82db4177cf24-kube-api-access-wjf69\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.464890 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.464875 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mg7d\" (UniqueName: \"kubernetes.io/projected/efe11c10-4c93-48fc-a213-0faaf29b5d11-kube-api-access-5mg7d\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.469452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.469432 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zgz\" (UniqueName: \"kubernetes.io/projected/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-kube-api-access-n8zgz\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.476142 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.476117 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g76\" (UniqueName: \"kubernetes.io/projected/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-kube-api-access-m9g76\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.476374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.476358 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsz49\" (UniqueName: \"kubernetes.io/projected/6e60683e-1ea5-4951-aae1-c76f68ef297a-kube-api-access-xsz49\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.480640 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.480621 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchwj\" (UniqueName: \"kubernetes.io/projected/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-kube-api-access-wchwj\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.480788 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.480772 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629cq\" (UniqueName: \"kubernetes.io/projected/80ef5a23-6398-4f49-b7f3-09f18b3473ca-kube-api-access-629cq\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.483653 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483635 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.483735 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483669 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.483735 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483710 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483735 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483744 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-node-log\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483768 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483792 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483808 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-bin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.483839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483822 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-hostroot\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483843 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-netns\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483874 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483894 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483911 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483916 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-socket-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483926 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483930 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-cnibin\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483950 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-etc-kubernetes\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483969 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-cni-multus\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483969 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483978 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-registration-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483996 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.484004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.483999 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-ovn\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484015 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484037 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484054 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4dc42eb-2297-455c-a982-d1f42131323c-crio-socket\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484062 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484067 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-k8s-cni-cncf-io\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484082 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484091 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484107 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-log-socket\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484112 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484118 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484140 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484145 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-system-cni-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484155 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484169 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484189 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484205 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-systemd-units\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484220 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60683e-1ea5-4951-aae1-c76f68ef297a-host\") pod \"node-ca-jk2pd\" (UID: \"6e60683e-1ea5-4951-aae1-c76f68ef297a\") " pod="openshift-image-registry/node-ca-jk2pd" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484229 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-etc-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484220 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484262 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484269 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-var-lib-kubelet\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484286 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484296 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-var-lib-kubelet\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484306 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484321 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.484345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484337 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484362 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-var-lib-openvswitch\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484364 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484393 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484403 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-system-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.484418 4524 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484435 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-slash\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484408 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484478 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-device-dir\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484487 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484453 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484479 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.484497 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.984483505 +0000 UTC m=+1.828155128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484568 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484600 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484625 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-systemd\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484627 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484651 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484653 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484679 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80ef5a23-6398-4f49-b7f3-09f18b3473ca-hosts-file\") pod \"node-resolver-hkb4l\" (UID: \"80ef5a23-6398-4f49-b7f3-09f18b3473ca\") " pod="openshift-dns/node-resolver-hkb4l" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484680 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484719 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484731 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1d03ece-45f6-4355-8414-729003908b01-host-slash\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484744 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484762 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-host\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484780 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484792 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484805 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-cni-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484798 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-multus-certs\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484806 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484841 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484858 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484865 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-sys-fs\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484877 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484891 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-dbus\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484903 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484916 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysconfig\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484923 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-kubernetes\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484927 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484962 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.484988 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485005 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-lib-modules\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485012 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485028 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-host-run-netns\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485034 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-sysctl-conf\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485045 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-kubelet\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485057 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-run\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485087 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485115 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485136 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-socket-dir-parent\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485140 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485161 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485176 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485179 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-run-systemd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485190 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485200 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-bin\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485203 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1124e17-db0a-42d9-8be6-82db4177cf24-multus-conf-dir\") pod \"multus-wbfff\" (UID: \"e1124e17-db0a-42d9-8be6-82db4177cf24\") " pod="openshift-multus/multus-wbfff" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485210 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485219 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-cni-netd\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485232 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-sys\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485247 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485267 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485272 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a1efff8-ab0b-4bde-841c-cce83e4ad6c7-etc-modprobe-d\") pod \"tuned-nvffm\" (UID: \"3a1efff8-ab0b-4bde-841c-cce83e4ad6c7\") " pod="openshift-cluster-node-tuning-operator/tuned-nvffm" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485284 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485290 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0525652-ce3f-4898-9be8-066de957058e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485323 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdfc0cb5-a33f-488c-974e-75c43659c08a-kubelet-config\") pod \"global-pull-secret-syncer-jqblf\" (UID: \"cdfc0cb5-a33f-488c-974e-75c43659c08a\") " pod="kube-system/global-pull-secret-syncer-jqblf" Oct 30 19:41:46.485400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.485330 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efe11c10-4c93-48fc-a213-0faaf29b5d11-cnibin\") pod \"multus-additional-cni-plugins-lg47k\" (UID: \"efe11c10-4c93-48fc-a213-0faaf29b5d11\") " pod="openshift-multus/multus-additional-cni-plugins-lg47k" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503224 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503223 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503269 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503279 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503291 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503301 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503311 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503293 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503333 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503335 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:41:46.503343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503344 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503372 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503527 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503543 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503571 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503608 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503658 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503661 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503672 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503716 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.503882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503795 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:41:46.504182 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503902 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.504182 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.503936 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:41:46.504559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.504544 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4dc42eb-2297-455c-a982-d1f42131323c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.506770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.506753 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qblk\" (UniqueName: \"kubernetes.io/projected/24b7a886-5347-4494-a757-db73e0177366-kube-api-access-6qblk\") pod \"migrator-66df5d496b-tk7kh\" (UID: \"24b7a886-5347-4494-a757-db73e0177366\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tk7kh" Oct 30 19:41:46.509772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.507097 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtf7\" (UniqueName: \"kubernetes.io/projected/d1d03ece-45f6-4355-8414-729003908b01-kube-api-access-wbtf7\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.509772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.507319 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8q8\" (UniqueName: \"kubernetes.io/projected/6e5c9fe7-ca03-478a-85ad-1a6e4afb4327-kube-api-access-vt8q8\") pod \"aws-ebs-csi-driver-node-b9tt2\" (UID: \"6e5c9fe7-ca03-478a-85ad-1a6e4afb4327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9tt2" Oct 30 19:41:46.509772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.507357 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzqr8\" (UniqueName: \"kubernetes.io/projected/260bbf80-e9c9-4264-8983-7bf37d39a563-kube-api-access-wzqr8\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.509772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.507363 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb6fk\" (UniqueName: \"kubernetes.io/projected/dd3626c0-00b5-4039-a06c-9537f72283bc-kube-api-access-jb6fk\") pod \"network-check-target-tdkh8\" (UID: \"dd3626c0-00b5-4039-a06c-9537f72283bc\") " pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:46.509772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.507518 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.510253 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.510233 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqqc\" (UniqueName: \"kubernetes.io/projected/b4dc42eb-2297-455c-a982-d1f42131323c-kube-api-access-sgqqc\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:46.511158 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.511142 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rqg\" (UniqueName: \"kubernetes.io/projected/24f786ff-dc91-4035-96cc-2e840a7fcae5-kube-api-access-x8rqg\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:46.511295 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.511282 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1d03ece-45f6-4355-8414-729003908b01-iptables-alerter-script\") pod \"iptables-alerter-j2p4l\" (UID: \"d1d03ece-45f6-4355-8414-729003908b01\") " pod="openshift-network-operator/iptables-alerter-j2p4l" Oct 30 19:41:46.512038 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.512023 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-config-volume\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:46.513185 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.513143 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.513270 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.513184 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-env-overrides\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.513270 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.513247 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-script-lib\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.513270 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.513246 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0525652-ce3f-4898-9be8-066de957058e-ovnkube-config\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.513270 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.513247 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-cabundle\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.519223 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.519206 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:41:46.519223 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.519204 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:41:46.519364 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.519327 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:41:46.520484 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.520467 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"periodic-gathering-h7q9z-4p4kw\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:41:46.520620 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.520608 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/260bbf80-e9c9-4264-8983-7bf37d39a563-signing-key\") pod \"service-ca-5fffc97bdf-g9npz\" (UID: \"260bbf80-e9c9-4264-8983-7bf37d39a563\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" Oct 30 19:41:46.525671 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525649 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:41:46.525770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525688 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:41:46.525770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525760 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9h48w\"" Oct 30 19:41:46.525886 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525759 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\"" Oct 30 19:41:46.525886 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525838 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:41:46.525886 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525845 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525905 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525929 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525945 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hpq6h\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525950 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v4lmq\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525946 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525954 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525974 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\"" Oct 30 19:41:46.526016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.525981 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\"" Oct 30 19:41:46.527429 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.527414 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllq9\" (UniqueName: \"kubernetes.io/projected/e0525652-ce3f-4898-9be8-066de957058e-kube-api-access-fllq9\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.532480 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.532464 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0525652-ce3f-4898-9be8-066de957058e-ovn-node-metrics-cert\") pod \"ovnkube-node-g8v6z\" (UID: \"e0525652-ce3f-4898-9be8-066de957058e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:46.532600 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.532586 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f786ff-dc91-4035-96cc-2e840a7fcae5-cert\") pod \"ingress-canary-k8hkx\" (UID: \"24f786ff-dc91-4035-96cc-2e840a7fcae5\") " pod="openshift-ingress-canary/ingress-canary-k8hkx" Oct 30 19:41:46.534690 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.534675 4524 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:46.534760 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.534750 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:47.034737927 +0000 UTC m=+1.878409535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:46.534809 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.534681 4524 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:46.534809 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.534796 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:47.034787106 +0000 UTC m=+1.878458729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:46.657926 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.657909 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:41:46.884742 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.884712 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.988798 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:46.988745 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:46.988891 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.988847 4524 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:46.988925 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:46.988904 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:47.988891698 +0000 UTC m=+2.832563304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:47.089418 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:47.089395 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:47.089499 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:47.089440 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:47.089543 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.089516 4524 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:47.089543 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.089517 4524 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:47.089608 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.089556 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.089545507 +0000 UTC m=+2.933217114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:47.089608 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.089570 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.089563347 +0000 UTC m=+2.933234954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:47.995256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:47.995160 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:47.995689 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.995310 4524 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:47.995689 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:47.995415 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:49.995393345 +0000 UTC m=+4.839064955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:48.096318 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.096283 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:48.096561 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.096345 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:48.096561 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:48.096430 4524 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:48.096561 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:48.096446 4524 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:48.096561 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:48.096497 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:50.096482942 +0000 UTC m=+4.940154549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:48.096561 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:48.096513 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:50.096503489 +0000 UTC m=+4.940175096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:48.340630 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.340550 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" event={"ID":"3bf076c5-f276-4191-9240-8968d8b20978","Type":"ContainerStarted","Data":"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150"} Oct 30 19:41:48.551360 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.551335 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:48.727340 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.727311 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.867636 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:48.867603 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:49.053539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:49.053466 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:49.179827 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:49.179800 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8v6z" Oct 30 19:41:50.009079 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:50.009041 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:50.009257 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.009216 4524 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:50.009327 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.009291 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:54.00927603 +0000 UTC m=+8.852947643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:50.109650 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:50.109616 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:50.110034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:50.109673 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:50.110034 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.109787 4524 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:50.110034 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.109804 4524 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:41:50.110034 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.109847 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:54.109830772 +0000 UTC m=+8.953502396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:50.110034 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:50.109882 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls podName:ae6f73a8-cc0c-4c3a-b771-90434a04fc32 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:54.109864402 +0000 UTC m=+8.953536011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls") pod "dns-default-78n2p" (UID: "ae6f73a8-cc0c-4c3a-b771-90434a04fc32") : secret "dns-default-metrics-tls" not found Oct 30 19:41:50.347511 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:50.347439 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-g9npz" event={"ID":"260bbf80-e9c9-4264-8983-7bf37d39a563","Type":"ContainerStarted","Data":"affaa1803df3f65574c92f0fb218002e5bb53b02f838053d1b9f69f641dfd317"} Oct 30 19:41:51.414039 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:51.414004 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.862021 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:51.861940 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.125191 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:52.125113 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.465626 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:52.465594 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.705687 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:52.705652 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.902102 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:52.902069 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hkb4l_80ef5a23-6398-4f49-b7f3-09f18b3473ca/dns-node-resolver/0.log" Oct 30 19:41:53.336160 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:53.336085 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.545170 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:53.545138 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:41:53.580941 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:53.580911 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.713430 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:53.713410 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jk2pd_6e60683e-1ea5-4951-aae1-c76f68ef297a/node-ca/0.log" Oct 30 19:41:53.832826 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:53.832796 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.039818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.039746 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:41:54.039949 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:54.039886 4524 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:41:54.039949 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:54.039947 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs podName:6eca3fa5-f093-4f9a-a7e7-d19fff01690f nodeName:}" failed. No retries permitted until 2026-10-30 19:42:02.039930941 +0000 UTC m=+16.883602563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs") pod "network-metrics-daemon-4tzpr" (UID: "6eca3fa5-f093-4f9a-a7e7-d19fff01690f") : secret "metrics-daemon-secret" not found Oct 30 19:41:54.078498 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.078469 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.140537 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.140509 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:54.140684 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.140558 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:41:54.140684 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:54.140648 4524 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:41:54.140785 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:41:54.140715 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls podName:b4dc42eb-2297-455c-a982-d1f42131323c nodeName:}" failed. No retries permitted until 2026-10-30 19:42:02.140685824 +0000 UTC m=+16.984357430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9tj4" (UID: "b4dc42eb-2297-455c-a982-d1f42131323c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:41:54.140938 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.140918 4524 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:41:54.142941 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.142923 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae6f73a8-cc0c-4c3a-b771-90434a04fc32-metrics-tls\") pod \"dns-default-78n2p\" (UID: \"ae6f73a8-cc0c-4c3a-b771-90434a04fc32\") " pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:54.156989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.156969 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:54.331468 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.331391 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.503206 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.503177 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8hkx_24f786ff-dc91-4035-96cc-2e840a7fcae5/serve-healthcheck-canary/0.log" Oct 30 19:41:54.529874 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.529850 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78n2p"] Oct 30 19:41:54.533576 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:41:54.533549 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f73a8_cc0c_4c3a_b771_90434a04fc32.slice/crio-f1e7c45ae1b59a15d938726a6a595351df263f7a0e2569faaf35da87ed90ffa2 WatchSource:0}: Error finding container f1e7c45ae1b59a15d938726a6a595351df263f7a0e2569faaf35da87ed90ffa2: Status 404 returned error can't find the container with id f1e7c45ae1b59a15d938726a6a595351df263f7a0e2569faaf35da87ed90ffa2 Oct 30 19:41:54.796380 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.796345 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.903077 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:54.903051 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tk7kh_24b7a886-5347-4494-a757-db73e0177366/migrator/0.log" Oct 30 19:41:55.351797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:55.351759 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.362333 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:55.362298 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78n2p" event={"ID":"ae6f73a8-cc0c-4c3a-b771-90434a04fc32","Type":"ContainerStarted","Data":"f1e7c45ae1b59a15d938726a6a595351df263f7a0e2569faaf35da87ed90ffa2"} Oct 30 19:41:55.608407 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:55.608252 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.864223 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:55.864115 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.130647 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:56.130576 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.377539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:56.377510 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.343090 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.343023 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.369068 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.369039 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78n2p" event={"ID":"ae6f73a8-cc0c-4c3a-b771-90434a04fc32","Type":"ContainerStarted","Data":"43d2c8b7432293a31f5227866049227663fa1fb02964c202ddf6362aca466305"} Oct 30 19:41:57.369199 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.369081 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78n2p" event={"ID":"ae6f73a8-cc0c-4c3a-b771-90434a04fc32","Type":"ContainerStarted","Data":"f6f6c8a5a029a7d5ff1d255d61b1691ba8fe1cb5ea7dffbbe4203269334937d0"} Oct 30 19:41:57.369199 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.369153 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-78n2p" Oct 30 19:41:57.580423 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.580393 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.896299 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:57.896263 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.172936 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:58.172853 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.456027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:59.455997 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.794128 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:41:59.794052 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.127379 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:00.127308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.417485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:00.417451 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.397395 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:01.397363 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.643611 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:01.643573 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.897454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:01.897424 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:02.098982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.098955 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:42:02.101187 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.101156 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3fa5-f093-4f9a-a7e7-d19fff01690f-metrics-certs\") pod \"network-metrics-daemon-4tzpr\" (UID: \"6eca3fa5-f093-4f9a-a7e7-d19fff01690f\") " pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:42:02.166345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.166276 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:02.199755 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.199735 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:42:02.201815 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.201799 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4dc42eb-2297-455c-a982-d1f42131323c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9tj4\" (UID: \"b4dc42eb-2297-455c-a982-d1f42131323c\") " pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:42:02.249992 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.249972 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tzpr" Oct 30 19:42:02.257958 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.257934 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9tj4" Oct 30 19:42:02.435287 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.435221 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:02.635331 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.635296 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tzpr"] Oct 30 19:42:02.666929 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:02.666904 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eca3fa5_f093_4f9a_a7e7_d19fff01690f.slice/crio-432d4be3dee2526fc19644ce0e51a98fdaed292747e67ca7c172fc539048bd4c WatchSource:0}: Error finding container 432d4be3dee2526fc19644ce0e51a98fdaed292747e67ca7c172fc539048bd4c: Status 404 returned error can't find the container with id 432d4be3dee2526fc19644ce0e51a98fdaed292747e67ca7c172fc539048bd4c Oct 30 19:42:02.684040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:02.684016 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dc42eb_2297_455c_a982_d1f42131323c.slice/crio-6e6cc99878094af089c9b96cce2cd6cdbae6e1df952c28d30314ce147c52325c WatchSource:0}: Error finding container 6e6cc99878094af089c9b96cce2cd6cdbae6e1df952c28d30314ce147c52325c: Status 404 returned error can't find the container with id 6e6cc99878094af089c9b96cce2cd6cdbae6e1df952c28d30314ce147c52325c Oct 30 19:42:02.685360 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:02.685315 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9tj4"] Oct 30 19:42:03.390031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:03.389996 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9tj4" event={"ID":"b4dc42eb-2297-455c-a982-d1f42131323c","Type":"ContainerStarted","Data":"83dbdeaf3a11427429ee67e2f011fd585896b564f6294e66786fa2560b13d785"} Oct 30 19:42:03.390208 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:03.390075 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9tj4" event={"ID":"b4dc42eb-2297-455c-a982-d1f42131323c","Type":"ContainerStarted","Data":"6e6cc99878094af089c9b96cce2cd6cdbae6e1df952c28d30314ce147c52325c"} Oct 30 19:42:03.391027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:03.391003 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tzpr" event={"ID":"6eca3fa5-f093-4f9a-a7e7-d19fff01690f","Type":"ContainerStarted","Data":"432d4be3dee2526fc19644ce0e51a98fdaed292747e67ca7c172fc539048bd4c"} Oct 30 19:42:03.818614 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:03.818585 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.296492 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:04.296458 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.396206 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:04.396176 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9tj4" event={"ID":"b4dc42eb-2297-455c-a982-d1f42131323c","Type":"ContainerStarted","Data":"c18745cce4323f85b503082f05392bf5afe487878daf4d1f4476a0c876942d6f"} Oct 30 19:42:04.397916 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:04.397893 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tzpr" event={"ID":"6eca3fa5-f093-4f9a-a7e7-d19fff01690f","Type":"ContainerStarted","Data":"89116a698ae211d57e4994cb353cf74a689f88df140514bfcc47290acae49a1f"} Oct 30 19:42:04.398022 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:04.397934 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tzpr" event={"ID":"6eca3fa5-f093-4f9a-a7e7-d19fff01690f","Type":"ContainerStarted","Data":"79975526b5d5fcbbbedb7caae6fbe2bba8db0d20f117e16ddec08bbf8d69780a"} Oct 30 19:42:05.163498 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:05.163457 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.729713 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:05.729671 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.035031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:06.034959 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.338594 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:06.338521 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.410291 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:06.410258 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9tj4" event={"ID":"b4dc42eb-2297-455c-a982-d1f42131323c","Type":"ContainerStarted","Data":"c1abd03762b2dbc810ee1865e8c2c2ac544a4fd2ff70c573c1eded3bff94185a"} Oct 30 19:42:06.851349 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:06.851318 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.106050 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:07.105983 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.375174 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:07.375109 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78n2p" Oct 30 19:42:07.558020 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:07.557989 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h2ktl" Oct 30 19:42:11.332452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.332414 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47"] Oct 30 19:42:11.334805 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.334790 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:11.393491 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.393464 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 19:42:11.442776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.442752 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gtf8f\"" Oct 30 19:42:11.457400 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.457375 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cf5d827-96a6-4b4b-a285-7021415f5b65-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-l2n47\" (UID: \"6cf5d827-96a6-4b4b-a285-7021415f5b65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:11.458540 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.458513 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64cc85466f-pz5ct"] Oct 30 19:42:11.460460 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.460445 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.534168 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.534133 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:42:11.534282 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.534189 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:42:11.534282 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.534257 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpdkk\"" Oct 30 19:42:11.558975 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.558954 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/794a71eb-4294-40d4-a854-6f002608c6aa-ca-trust-extracted\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.558979 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-registry-certificates\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559009 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-trusted-ca\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559026 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-installation-pull-secrets\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559163 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-image-registry-private-configuration\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559215 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cf5d827-96a6-4b4b-a285-7021415f5b65-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-l2n47\" (UID: \"6cf5d827-96a6-4b4b-a285-7021415f5b65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:11.559345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559326 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqps\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-kube-api-access-nrqps\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559411 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-bound-sa-token\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.559480 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.559446 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-registry-tls\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.561570 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.561554 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cf5d827-96a6-4b4b-a285-7021415f5b65-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-l2n47\" (UID: \"6cf5d827-96a6-4b4b-a285-7021415f5b65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:11.592965 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.592912 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:42:11.592965 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.592939 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:42:11.631295 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.631274 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.636713 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.636672 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cc85466f-pz5ct"] Oct 30 19:42:11.636814 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.636795 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47"] Oct 30 19:42:11.644145 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.644128 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:11.660069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660046 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-trusted-ca\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660169 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660081 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-installation-pull-secrets\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660169 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660110 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-image-registry-private-configuration\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660169 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660147 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqps\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-kube-api-access-nrqps\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660324 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660173 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-bound-sa-token\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660324 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660211 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-registry-tls\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660324 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660238 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/794a71eb-4294-40d4-a854-6f002608c6aa-ca-trust-extracted\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.660938 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.660890 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/794a71eb-4294-40d4-a854-6f002608c6aa-ca-trust-extracted\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.661360 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.661341 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-registry-certificates\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.661626 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.661416 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-trusted-ca\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.662198 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.662174 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/794a71eb-4294-40d4-a854-6f002608c6aa-registry-certificates\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.662953 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.662934 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-image-registry-private-configuration\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.663060 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.663036 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/794a71eb-4294-40d4-a854-6f002608c6aa-installation-pull-secrets\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.663760 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.663741 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-registry-tls\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.724246 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.724220 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqps\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-kube-api-access-nrqps\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.726394 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.726375 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/794a71eb-4294-40d4-a854-6f002608c6aa-bound-sa-token\") pod \"image-registry-64cc85466f-pz5ct\" (UID: \"794a71eb-4294-40d4-a854-6f002608c6aa\") " pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.769443 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.769416 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:11.965239 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:11.965185 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.026583 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.026559 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47"] Oct 30 19:42:12.029594 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:12.029569 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf5d827_96a6_4b4b_a285_7021415f5b65.slice/crio-f0cfc12ba728f5ee298355fb4dc0c2d4d9ccecc7fa34c8a3894d0c0840fbf39a WatchSource:0}: Error finding container f0cfc12ba728f5ee298355fb4dc0c2d4d9ccecc7fa34c8a3894d0c0840fbf39a: Status 404 returned error can't find the container with id f0cfc12ba728f5ee298355fb4dc0c2d4d9ccecc7fa34c8a3894d0c0840fbf39a Oct 30 19:42:12.152570 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.152532 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cc85466f-pz5ct"] Oct 30 19:42:12.157585 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:12.157559 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794a71eb_4294_40d4_a854_6f002608c6aa.slice/crio-739e941149dfdc0ce2173f753d1057b3ddca2beabb7337bb5cb8da631b24a180 WatchSource:0}: Error finding container 739e941149dfdc0ce2173f753d1057b3ddca2beabb7337bb5cb8da631b24a180: Status 404 returned error can't find the container with id 739e941149dfdc0ce2173f753d1057b3ddca2beabb7337bb5cb8da631b24a180 Oct 30 19:42:12.191185 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.191163 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.428894 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.428855 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" event={"ID":"794a71eb-4294-40d4-a854-6f002608c6aa","Type":"ContainerStarted","Data":"95ffebea626657b8a03d252801ec829f79090c7d7e0a78340728a01e1dab1001"} Oct 30 19:42:12.429322 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.428915 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" event={"ID":"794a71eb-4294-40d4-a854-6f002608c6aa","Type":"ContainerStarted","Data":"739e941149dfdc0ce2173f753d1057b3ddca2beabb7337bb5cb8da631b24a180"} Oct 30 19:42:12.429322 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.428942 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:12.430183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.430157 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" event={"ID":"6cf5d827-96a6-4b4b-a285-7021415f5b65","Type":"ContainerStarted","Data":"f0cfc12ba728f5ee298355fb4dc0c2d4d9ccecc7fa34c8a3894d0c0840fbf39a"} Oct 30 19:42:12.553501 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:12.553298 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" podStartSLOduration=1.5532729889999999 podStartE2EDuration="1.553272989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:12.553139882 +0000 UTC m=+27.396811514" watchObservedRunningTime="2026-10-30 19:42:12.553272989 +0000 UTC m=+27.396944619" Oct 30 19:42:12.783074 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:42:12.783003 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:42:13.548181 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:13.548161 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tdkh8" Oct 30 19:42:14.438310 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:14.438273 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" event={"ID":"6cf5d827-96a6-4b4b-a285-7021415f5b65","Type":"ContainerStarted","Data":"33d84ae9b60069319a69e4e93af292f18aaa146f0409be94ffc57e310a8ccca6"} Oct 30 19:42:14.438485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:14.438469 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:14.443189 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:14.443168 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" Oct 30 19:42:14.555511 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:14.555464 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-l2n47" podStartSLOduration=2.084865599 podStartE2EDuration="3.555451364s" totalImagesPullingTime="1.470585765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:14.554596158 +0000 UTC m=+29.398267789" watchObservedRunningTime="2026-10-30 19:42:14.555451364 +0000 UTC m=+29.399122992" Oct 30 19:42:15.152532 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.152500 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.395427 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.395388 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.518934 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.518845 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk"] Oct 30 19:42:15.539916 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.539886 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.582413 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.582385 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk"] Oct 30 19:42:15.631682 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.631660 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 19:42:15.631818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.631647 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.641875 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.641855 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:42:15.681631 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.681608 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-smq4d\"" Oct 30 19:42:15.681631 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.681608 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:42:15.681789 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.681618 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:42:15.682045 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.682030 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 19:42:15.688507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.688488 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.688545 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.688516 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.688545 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.688537 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5lp\" (UniqueName: \"kubernetes.io/projected/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-kube-api-access-lp5lp\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.688638 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.688558 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.789693 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.789638 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.789693 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.789688 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.789843 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.789805 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.789876 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.789843 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5lp\" (UniqueName: \"kubernetes.io/projected/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-kube-api-access-lp5lp\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.790295 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.790274 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.792042 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.792022 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.792042 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.792035 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.850404 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.850383 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5lp\" (UniqueName: \"kubernetes.io/projected/28f6fd9e-fe8a-4e23-87d8-38a68e9d1448-kube-api-access-lp5lp\") pod \"prometheus-operator-9cd89b74c-s9zjk\" (UID: \"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:15.877038 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:15.877017 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" Oct 30 19:42:16.268302 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:16.268275 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk"] Oct 30 19:42:16.271689 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:16.271666 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f6fd9e_fe8a_4e23_87d8_38a68e9d1448.slice/crio-417957f7538114d32424b823bf86bafe5d5dd597882435143408eb4cc24f55a8 WatchSource:0}: Error finding container 417957f7538114d32424b823bf86bafe5d5dd597882435143408eb4cc24f55a8: Status 404 returned error can't find the container with id 417957f7538114d32424b823bf86bafe5d5dd597882435143408eb4cc24f55a8 Oct 30 19:42:16.445029 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:16.444996 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" event={"ID":"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448","Type":"ContainerStarted","Data":"417957f7538114d32424b823bf86bafe5d5dd597882435143408eb4cc24f55a8"} Oct 30 19:42:16.892337 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:16.892301 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:17.151270 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:17.151201 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.631164 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:18.631131 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.899194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:18.899164 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.143251 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.143217 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.461273 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.461239 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" event={"ID":"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448","Type":"ContainerStarted","Data":"fd3d88f85f5d2adacb6321a54a65216d50451d11c442c8487874794ba853f4c7"} Oct 30 19:42:19.461423 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.461289 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" event={"ID":"28f6fd9e-fe8a-4e23-87d8-38a68e9d1448","Type":"ContainerStarted","Data":"8fcff326bea8678560f75eff5370458848b6818e1c3b91d47df2659f9a9f8613"} Oct 30 19:42:19.558925 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.558897 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.584057 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.584015 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9zjk" podStartSLOduration=2.346239625 podStartE2EDuration="4.584002985s" totalImagesPullingTime="2.23776336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:19.582440709 +0000 UTC m=+34.426112339" watchObservedRunningTime="2026-10-30 19:42:19.584002985 +0000 UTC m=+34.427674608" Oct 30 19:42:19.819086 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:19.819024 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.034085 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:20.034055 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.345335 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:20.345307 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.614136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:20.614046 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:21.282637 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:21.282602 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.418383 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:22.418350 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.660159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:22.660122 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.916944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:22.916913 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.172638 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.172559 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.192337 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.192303 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dc8mn"] Oct 30 19:42:23.195439 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.195423 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.269020 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.268974 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:42:23.305607 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.305574 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:42:23.305779 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.305714 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:42:23.305908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.305895 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fn2xq\"" Oct 30 19:42:23.342717 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.342682 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-tls\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.342872 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.342734 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-root\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.342872 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.342759 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-metrics-client-ca\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.342872 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.342853 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzqr\" (UniqueName: \"kubernetes.io/projected/f313d87a-d788-4a3e-9e2a-43ea389e9a54-kube-api-access-7jzqr\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.343034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.342889 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.343034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.343013 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.343123 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.343056 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-textfile\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.343123 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.343108 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-wtmp\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.343209 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.343129 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-sys\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.392375 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.392338 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.444213 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444141 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444213 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444181 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-textfile\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444216 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-wtmp\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444242 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-sys\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444274 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-tls\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444351 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-root\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444391 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-metrics-client-ca\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444355 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-sys\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444403 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-wtmp\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444395 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f313d87a-d788-4a3e-9e2a-43ea389e9a54-root\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444426 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzqr\" (UniqueName: \"kubernetes.io/projected/f313d87a-d788-4a3e-9e2a-43ea389e9a54-kube-api-access-7jzqr\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444466 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.444721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444638 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-textfile\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.445167 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.444946 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-metrics-client-ca\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.445167 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.445075 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.447169 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.447146 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.447262 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.447146 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f313d87a-d788-4a3e-9e2a-43ea389e9a54-node-exporter-tls\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.506156 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.506125 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzqr\" (UniqueName: \"kubernetes.io/projected/f313d87a-d788-4a3e-9e2a-43ea389e9a54-kube-api-access-7jzqr\") pod \"node-exporter-dc8mn\" (UID: \"f313d87a-d788-4a3e-9e2a-43ea389e9a54\") " pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.506993 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.506969 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dc8mn" Oct 30 19:42:23.517510 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:23.517482 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf313d87a_d788_4a3e_9e2a_43ea389e9a54.slice/crio-13e8f1d4a623f590cd290079c03ec8c2f39c60919d5618eef8a1e1094e32c5e8 WatchSource:0}: Error finding container 13e8f1d4a623f590cd290079c03ec8c2f39c60919d5618eef8a1e1094e32c5e8: Status 404 returned error can't find the container with id 13e8f1d4a623f590cd290079c03ec8c2f39c60919d5618eef8a1e1094e32c5e8 Oct 30 19:42:23.636450 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.636418 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.890252 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:23.890228 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:24.478489 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:24.478456 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dc8mn" event={"ID":"f313d87a-d788-4a3e-9e2a-43ea389e9a54","Type":"ContainerStarted","Data":"13e8f1d4a623f590cd290079c03ec8c2f39c60919d5618eef8a1e1094e32c5e8"} Oct 30 19:42:24.492040 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:24.492014 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:24.754008 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:24.753930 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.002182 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:25.002150 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.482861 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:25.482829 4524 generic.go:356] "Generic (PLEG): container finished" podID="f313d87a-d788-4a3e-9e2a-43ea389e9a54" containerID="beea764343975e267e65b29dd0ecb73c51aed293b46a0663ca6f5e509be6c162" exitCode=0 Oct 30 19:42:25.483242 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:25.482901 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dc8mn" event={"ID":"f313d87a-d788-4a3e-9e2a-43ea389e9a54","Type":"ContainerDied","Data":"beea764343975e267e65b29dd0ecb73c51aed293b46a0663ca6f5e509be6c162"} Oct 30 19:42:25.600135 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:25.600113 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.892110 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:25.892080 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.191374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:26.191290 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.491989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:26.491890 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dc8mn" event={"ID":"f313d87a-d788-4a3e-9e2a-43ea389e9a54","Type":"ContainerStarted","Data":"af7baacd179e2d9a2851737d59abb8b7b2267bfc4f9a77c980394e07a8f55f28"} Oct 30 19:42:26.491989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:26.491964 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dc8mn" event={"ID":"f313d87a-d788-4a3e-9e2a-43ea389e9a54","Type":"ContainerStarted","Data":"d707dc35e3aad18236ef42896888da1a688cd6e253de4155ab9393c7db4ac1ce"} Oct 30 19:42:26.671822 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:26.671774 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dc8mn" podStartSLOduration=2.170282219 podStartE2EDuration="3.671761998s" totalImagesPullingTime="1.434177809s" totalInitContainerRuntime="67.30197ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.670977446 +0000 UTC m=+41.514649084" watchObservedRunningTime="2026-10-30 19:42:26.671761998 +0000 UTC m=+41.515433626" Oct 30 19:42:27.757563 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:27.757507 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.068884 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:28.068802 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.325207 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:28.325131 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.628817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:28.628741 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.882374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:28.882295 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:29.192563 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:29.192525 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:32.167393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:32.167335 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:32.434588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:32.434506 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:33.054447 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:33.054412 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:33.437908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:33.437883 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64cc85466f-pz5ct" Oct 30 19:42:33.732177 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:33.732095 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.037508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:34.037407 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.446390 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:34.446354 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.681930 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:34.681899 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:36.991446 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:36.991398 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:37.273588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:37.273502 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.114341 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.114307 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:42:39.118128 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.118101 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.190892 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.190863 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:42:39.231410 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.231369 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:42:39.231410 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.231374 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-stjbd\"" Oct 30 19:42:39.231604 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.231418 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:42:39.231604 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.231418 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:42:39.231604 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.231482 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:42:39.240876 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.240853 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:42:39.241019 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.240941 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:42:39.257376 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257347 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257383 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257407 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257581 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257542 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpn9\" (UniqueName: \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257581 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257575 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257655 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257599 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.257689 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.257671 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.282329 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.282306 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:42:39.358908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.358883 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359051 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.358930 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359117 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359096 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359167 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359149 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359182 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359281 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359230 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpn9\" (UniqueName: \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359281 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359264 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359646 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359624 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359766 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359672 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359833 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359796 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.359915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.359895 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.361320 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.361301 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.361485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.361468 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.435831 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.435805 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpn9\" (UniqueName: \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\") pod \"console-8d65fbc65-njxqz\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:39.730547 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:39.730478 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:40.104805 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:40.104693 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:42:40.107173 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:40.107144 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751c631e_daed_4152_8e17_f5a87972edb9.slice/crio-503499bc64ac09e09fee3ef7e574525de8846f3f945a7d4fd74db6582a9655fc WatchSource:0}: Error finding container 503499bc64ac09e09fee3ef7e574525de8846f3f945a7d4fd74db6582a9655fc: Status 404 returned error can't find the container with id 503499bc64ac09e09fee3ef7e574525de8846f3f945a7d4fd74db6582a9655fc Oct 30 19:42:40.537628 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:40.537592 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d65fbc65-njxqz" event={"ID":"751c631e-daed-4152-8e17-f5a87972edb9","Type":"ContainerStarted","Data":"503499bc64ac09e09fee3ef7e574525de8846f3f945a7d4fd74db6582a9655fc"} Oct 30 19:42:42.789323 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:42:42.789275 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:42:43.550563 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:43.550527 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d65fbc65-njxqz" event={"ID":"751c631e-daed-4152-8e17-f5a87972edb9","Type":"ContainerStarted","Data":"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30"} Oct 30 19:42:43.703291 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:43.703114 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8d65fbc65-njxqz" podStartSLOduration=1.433288623 podStartE2EDuration="4.703097346s" totalImagesPullingTime="3.269808723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:43.702977413 +0000 UTC m=+58.546649042" watchObservedRunningTime="2026-10-30 19:42:43.703097346 +0000 UTC m=+58.546768976" Oct 30 19:42:43.757636 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:43.757608 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:43.977823 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:43.977744 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.861315 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:44.861277 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.079857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:45.079825 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:48.685830 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:48.685786 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:48.907620 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:48.907591 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.312249 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:49.312217 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.620959 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:49.620884 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.731464 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:49.731435 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:49.731848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:49.731488 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:49.736212 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:49.736194 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:50.575208 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:50.575177 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:42:51.373219 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.373190 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:42:51.378293 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.378275 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450557 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450528 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450567 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450590 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvr59\" (UniqueName: \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450691 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450840 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450732 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450840 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450758 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450910 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450840 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.450910 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.450865 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551185 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551152 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551185 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551190 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvr59\" (UniqueName: \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551377 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551236 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551377 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551333 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551469 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551376 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551469 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551408 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551469 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551435 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551616 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551492 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551672 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551624 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.551986 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.551957 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.552136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.552021 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.552136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.552120 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.554056 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.554033 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.554113 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.554075 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.582155 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.582130 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:42:51.583017 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.583000 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:42:51.592977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.592959 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.613104 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.613084 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvr59\" (UniqueName: \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\") pod \"console-6b86d5cf45-z9cw5\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:51.689380 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:51.689354 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:42:52.066346 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.066278 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:42:52.069518 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:42:52.069490 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7d9808_8a03_46f6_aa12_71218bde67d6.slice/crio-cc2e6461a8e75a51ead3af6c145bd34276b1fec1918405c6605169d3286cfe2c WatchSource:0}: Error finding container cc2e6461a8e75a51ead3af6c145bd34276b1fec1918405c6605169d3286cfe2c: Status 404 returned error can't find the container with id cc2e6461a8e75a51ead3af6c145bd34276b1fec1918405c6605169d3286cfe2c Oct 30 19:42:52.578735 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.578682 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b86d5cf45-z9cw5" event={"ID":"7d7d9808-8a03-46f6-aa12-71218bde67d6","Type":"ContainerStarted","Data":"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad"} Oct 30 19:42:52.579082 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.578761 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b86d5cf45-z9cw5" event={"ID":"7d7d9808-8a03-46f6-aa12-71218bde67d6","Type":"ContainerStarted","Data":"cc2e6461a8e75a51ead3af6c145bd34276b1fec1918405c6605169d3286cfe2c"} Oct 30 19:42:52.698316 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.698270 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b86d5cf45-z9cw5" podStartSLOduration=1.698256807 podStartE2EDuration="1.698256807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:52.697037658 +0000 UTC m=+67.540709286" watchObservedRunningTime="2026-10-30 19:42:52.698256807 +0000 UTC m=+67.541928435" Oct 30 19:42:52.742448 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.742422 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:52.958717 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:52.958667 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.143512 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:55.143480 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.423136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:55.423057 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:56.313288 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:56.313249 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:56.538223 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:56.538183 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.711047 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:57.711015 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.964574 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:57.964495 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.946736 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:42:59.946682 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.689850 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:01.689818 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:01.690256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:01.689872 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:01.694395 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:01.694373 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:02.057856 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.057787 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:43:02.105101 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.105075 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d694c6888-zhvph"] Oct 30 19:43:02.109988 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.109969 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.171199 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.171173 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d694c6888-zhvph"] Oct 30 19:43:02.231824 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.231791 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-service-ca\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.231824 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.231825 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/439ad812-ae62-4895-aa40-024c214a2eab-tmp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.231844 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-oauth-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.231921 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-console-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.231952 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grkp\" (UniqueName: \"kubernetes.io/projected/439ad812-ae62-4895-aa40-024c214a2eab-kube-api-access-5grkp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.232006 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-trusted-ca-bundle\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232184 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.232043 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.232184 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.232066 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-oauth-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332446 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332376 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332446 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332417 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-oauth-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332446 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332438 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-service-ca\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332446 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332453 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/439ad812-ae62-4895-aa40-024c214a2eab-tmp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332728 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332471 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-oauth-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332728 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332499 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-console-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332728 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332523 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grkp\" (UniqueName: \"kubernetes.io/projected/439ad812-ae62-4895-aa40-024c214a2eab-kube-api-access-5grkp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.332728 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.332545 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-trusted-ca-bundle\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.333028 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.333003 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/439ad812-ae62-4895-aa40-024c214a2eab-tmp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.333294 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.333270 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-service-ca\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.333363 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.333315 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-oauth-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.333413 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.333390 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-console-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.333469 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.333455 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/439ad812-ae62-4895-aa40-024c214a2eab-trusted-ca-bundle\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.334912 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.334894 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-serving-cert\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.335005 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.334933 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/439ad812-ae62-4895-aa40-024c214a2eab-console-oauth-config\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.396530 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.396509 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grkp\" (UniqueName: \"kubernetes.io/projected/439ad812-ae62-4895-aa40-024c214a2eab-kube-api-access-5grkp\") pod \"console-5d694c6888-zhvph\" (UID: \"439ad812-ae62-4895-aa40-024c214a2eab\") " pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.421646 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.421625 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:02.614814 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.614740 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:02.802090 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:02.802062 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d694c6888-zhvph"] Oct 30 19:43:02.805040 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:43:02.805016 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439ad812_ae62_4895_aa40_024c214a2eab.slice/crio-013a08bb45996faf6898bd4771af8599c9bc11552a6f27566f1ab8f5788e26aa WatchSource:0}: Error finding container 013a08bb45996faf6898bd4771af8599c9bc11552a6f27566f1ab8f5788e26aa: Status 404 returned error can't find the container with id 013a08bb45996faf6898bd4771af8599c9bc11552a6f27566f1ab8f5788e26aa Oct 30 19:43:03.617887 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:03.617850 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d694c6888-zhvph" event={"ID":"439ad812-ae62-4895-aa40-024c214a2eab","Type":"ContainerStarted","Data":"dc2630779053c9d9b490dd2b4da4e57bd595cd8e0d7c7dfce084462ece1aa772"} Oct 30 19:43:03.618087 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:03.617901 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d694c6888-zhvph" event={"ID":"439ad812-ae62-4895-aa40-024c214a2eab","Type":"ContainerStarted","Data":"013a08bb45996faf6898bd4771af8599c9bc11552a6f27566f1ab8f5788e26aa"} Oct 30 19:43:03.679384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:03.679347 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.763444 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:03.763393 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d694c6888-zhvph" podStartSLOduration=1.763381554 podStartE2EDuration="1.763381554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:03.743148722 +0000 UTC m=+78.586820351" watchObservedRunningTime="2026-10-30 19:43:03.763381554 +0000 UTC m=+78.607053182" Oct 30 19:43:03.894751 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:03.894723 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.312969 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.312937 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.558576 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.558534 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.598147 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.598080 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:43:11.598308 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.598274 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" podUID="3bf076c5-f276-4191-9240-8968d8b20978" containerName="insights-gathering" containerID="cri-o://158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150" gracePeriod=30 Oct 30 19:43:11.848662 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.848613 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:43:11.914302 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914279 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") pod \"3bf076c5-f276-4191-9240-8968d8b20978\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " Oct 30 19:43:11.914393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914336 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") pod \"3bf076c5-f276-4191-9240-8968d8b20978\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " Oct 30 19:43:11.914393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914360 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") pod \"3bf076c5-f276-4191-9240-8968d8b20978\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " Oct 30 19:43:11.914393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914386 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") pod \"3bf076c5-f276-4191-9240-8968d8b20978\" (UID: \"3bf076c5-f276-4191-9240-8968d8b20978\") " Oct 30 19:43:11.914639 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914614 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle" pod "3bf076c5-f276-4191-9240-8968d8b20978" (UID: "3bf076c5-f276-4191-9240-8968d8b20978"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:11.914731 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914713 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path" pod "3bf076c5-f276-4191-9240-8968d8b20978" (UID: "3bf076c5-f276-4191-9240-8968d8b20978"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:11.914865 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.914842 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle" pod "3bf076c5-f276-4191-9240-8968d8b20978" (UID: "3bf076c5-f276-4191-9240-8968d8b20978"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:11.916549 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:11.916527 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf" pod "3bf076c5-f276-4191-9240-8968d8b20978" (UID: "3bf076c5-f276-4191-9240-8968d8b20978"). InnerVolumeSpecName "kube-api-access-66fvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:12.015205 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.015169 4524 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3bf076c5-f276-4191-9240-8968d8b20978-archives-path\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:12.015205 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.015199 4524 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-trusted-ca-bundle\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:12.015375 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.015212 4524 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf076c5-f276-4191-9240-8968d8b20978-service-ca-bundle\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:12.015375 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.015226 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fvf\" (UniqueName: \"kubernetes.io/projected/3bf076c5-f276-4191-9240-8968d8b20978-kube-api-access-66fvf\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:12.421990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.421956 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:12.422482 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.422016 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:12.426721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.426680 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:12.648485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.648445 4524 generic.go:356] "Generic (PLEG): container finished" podID="3bf076c5-f276-4191-9240-8968d8b20978" containerID="158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150" exitCode=2 Oct 30 19:43:12.648659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.648488 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" Oct 30 19:43:12.648659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.648526 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" event={"ID":"3bf076c5-f276-4191-9240-8968d8b20978","Type":"ContainerDied","Data":"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150"} Oct 30 19:43:12.648659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.648579 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h7q9z-4p4kw" event={"ID":"3bf076c5-f276-4191-9240-8968d8b20978","Type":"ContainerDied","Data":"2c631ef2717ff78202b268554e500a589649d0b3b0c7535e702220f3dcdd2835"} Oct 30 19:43:12.648659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.648598 4524 scope.go:122] "RemoveContainer" containerID="158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150" Oct 30 19:43:12.652589 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.652570 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d694c6888-zhvph" Oct 30 19:43:12.656497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.656445 4524 scope.go:122] "RemoveContainer" containerID="158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150" Oct 30 19:43:12.656801 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:43:12.656779 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150\": container with ID starting with 158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150 not found: ID does not exist" containerID="158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150" Oct 30 19:43:12.656872 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.656808 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150"} err="failed to get container status \"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150\": rpc error: code = NotFound desc = could not find container \"158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150\": container with ID starting with 158fb01b073fdb61ee23d271e6696de8396d2a8089e2134fe374d29beec12150 not found: ID does not exist" Oct 30 19:43:12.784654 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:43:12.784583 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:43:12.859432 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.859402 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:43:12.935287 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.935258 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:43:12.937735 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:12.937694 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-h7q9z-4p4kw"] Oct 30 19:43:13.297542 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:13.297510 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf076c5-f276-4191-9240-8968d8b20978" path="/var/lib/kubelet/pods/3bf076c5-f276-4191-9240-8968d8b20978/volumes" Oct 30 19:43:16.963342 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:16.963306 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.181714 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:17.181671 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:28.636652 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:28.636587 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b86d5cf45-z9cw5" podUID="7d7d9808-8a03-46f6-aa12-71218bde67d6" containerName="console" containerID="cri-o://b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad" gracePeriod=15 Oct 30 19:43:28.887485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:28.887432 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:29.031759 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031723 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.031944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031769 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.031944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031822 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.031944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031848 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.031944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031875 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.031944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031903 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.032194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.031977 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\" (UniqueName: \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.032194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032010 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\") pod \"7d7d9808-8a03-46f6-aa12-71218bde67d6\" (UID: \"7d7d9808-8a03-46f6-aa12-71218bde67d6\") " Oct 30 19:43:29.032194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032022 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:29.032333 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032211 4524 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d7d9808-8a03-46f6-aa12-71218bde67d6-tmp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.032384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032305 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:29.032442 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032380 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:29.032442 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032412 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:29.032551 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.032534 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:29.034048 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.034014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:29.034159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.034106 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "kube-api-access-lvr59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:29.034159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.034111 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert" pod "7d7d9808-8a03-46f6-aa12-71218bde67d6" (UID: "7d7d9808-8a03-46f6-aa12-71218bde67d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:29.133311 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133277 4524 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133311 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133305 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvr59\" (UniqueName: \"kubernetes.io/projected/7d7d9808-8a03-46f6-aa12-71218bde67d6-kube-api-access-lvr59\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133311 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133316 4524 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-trusted-ca-bundle\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133521 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133325 4524 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-oauth-config\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133521 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133334 4524 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-config\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133521 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133342 4524 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d7d9808-8a03-46f6-aa12-71218bde67d6-console-serving-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.133521 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.133352 4524 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d9808-8a03-46f6-aa12-71218bde67d6-oauth-serving-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:29.704895 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.704864 4524 generic.go:356] "Generic (PLEG): container finished" podID="7d7d9808-8a03-46f6-aa12-71218bde67d6" containerID="b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad" exitCode=0 Oct 30 19:43:29.705354 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.704910 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b86d5cf45-z9cw5" Oct 30 19:43:29.705354 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.704946 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b86d5cf45-z9cw5" event={"ID":"7d7d9808-8a03-46f6-aa12-71218bde67d6","Type":"ContainerDied","Data":"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad"} Oct 30 19:43:29.705354 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.704997 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b86d5cf45-z9cw5" event={"ID":"7d7d9808-8a03-46f6-aa12-71218bde67d6","Type":"ContainerDied","Data":"cc2e6461a8e75a51ead3af6c145bd34276b1fec1918405c6605169d3286cfe2c"} Oct 30 19:43:29.705354 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.705022 4524 scope.go:122] "RemoveContainer" containerID="b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad" Oct 30 19:43:29.712624 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.712605 4524 scope.go:122] "RemoveContainer" containerID="b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad" Oct 30 19:43:29.712894 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:43:29.712873 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad\": container with ID starting with b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad not found: ID does not exist" containerID="b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad" Oct 30 19:43:29.712972 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.712908 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad"} err="failed to get container status \"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad\": rpc error: code = NotFound desc = could not find container \"b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad\": container with ID starting with b0e7acb3c228aac43bccac64b6cf0942eb445e9b5d652924ed50a9f55367e9ad not found: ID does not exist" Oct 30 19:43:29.869895 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.869866 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:43:29.920041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:29.920017 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b86d5cf45-z9cw5"] Oct 30 19:43:31.296669 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:31.296636 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7d9808-8a03-46f6-aa12-71218bde67d6" path="/var/lib/kubelet/pods/7d7d9808-8a03-46f6-aa12-71218bde67d6/volumes" Oct 30 19:43:34.544975 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:34.544932 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:37.877674 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:37.877606 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8d65fbc65-njxqz" podUID="751c631e-daed-4152-8e17-f5a87972edb9" containerName="console" containerID="cri-o://16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30" gracePeriod=15 Oct 30 19:43:38.127386 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.127361 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:43:38.294687 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294663 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294711 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294747 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294778 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294939 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294799 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294939 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294847 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\" (UniqueName: \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.294939 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294880 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\") pod \"751c631e-daed-4152-8e17-f5a87972edb9\" (UID: \"751c631e-daed-4152-8e17-f5a87972edb9\") " Oct 30 19:43:38.295077 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.294927 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:38.295136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.295093 4524 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/751c631e-daed-4152-8e17-f5a87972edb9-tmp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.295247 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.295216 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:38.295320 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.295259 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:38.295320 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.295275 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:38.296978 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.296947 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:38.297071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.296987 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:38.297071 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.297039 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9" pod "751c631e-daed-4152-8e17-f5a87972edb9" (UID: "751c631e-daed-4152-8e17-f5a87972edb9"). InnerVolumeSpecName "kube-api-access-hhpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:38.395497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395473 4524 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.395497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395494 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpn9\" (UniqueName: \"kubernetes.io/projected/751c631e-daed-4152-8e17-f5a87972edb9-kube-api-access-hhpn9\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.395605 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395506 4524 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-oauth-config\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.395605 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395516 4524 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-console-config\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.395605 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395525 4524 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751c631e-daed-4152-8e17-f5a87972edb9-oauth-serving-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.395605 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.395533 4524 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751c631e-daed-4152-8e17-f5a87972edb9-console-serving-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:38.735551 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.735524 4524 generic.go:356] "Generic (PLEG): container finished" podID="751c631e-daed-4152-8e17-f5a87972edb9" containerID="16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30" exitCode=0 Oct 30 19:43:38.735691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.735578 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d65fbc65-njxqz" Oct 30 19:43:38.735691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.735587 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d65fbc65-njxqz" event={"ID":"751c631e-daed-4152-8e17-f5a87972edb9","Type":"ContainerDied","Data":"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30"} Oct 30 19:43:38.735691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.735625 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d65fbc65-njxqz" event={"ID":"751c631e-daed-4152-8e17-f5a87972edb9","Type":"ContainerDied","Data":"503499bc64ac09e09fee3ef7e574525de8846f3f945a7d4fd74db6582a9655fc"} Oct 30 19:43:38.735691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.735644 4524 scope.go:122] "RemoveContainer" containerID="16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30" Oct 30 19:43:38.743904 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.743887 4524 scope.go:122] "RemoveContainer" containerID="16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30" Oct 30 19:43:38.744209 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:43:38.744190 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30\": container with ID starting with 16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30 not found: ID does not exist" containerID="16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30" Oct 30 19:43:38.744276 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.744218 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30"} err="failed to get container status \"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30\": rpc error: code = NotFound desc = could not find container \"16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30\": container with ID starting with 16b64f547a4509c2dd6a1aae7c8fa53e57fbf1fb45e86e25d21ff7fb70eaab30 not found: ID does not exist" Oct 30 19:43:38.900058 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.900032 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:43:38.949738 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:38.949715 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d65fbc65-njxqz"] Oct 30 19:43:39.296364 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:43:39.296329 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751c631e-daed-4152-8e17-f5a87972edb9" path="/var/lib/kubelet/pods/751c631e-daed-4152-8e17-f5a87972edb9/volumes" Oct 30 19:43:42.783197 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:43:42.783162 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:44:12.784429 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:44:12.784394 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:44:27.238295 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:44:27.238261 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.901814 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:44:34.901776 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:42.784852 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:44:42.784812 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:44:55.201859 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:44:55.201822 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.783649 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:45:12.783613 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:45:35.261805 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:45:35.261721 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.130915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:45:40.130875 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:42.784094 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:45:42.784059 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:45:43.024932 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:45:43.024898 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:43.233387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:45:43.233350 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:03.686653 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:46:03.686613 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:03.908681 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:46:03.908650 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:04.126677 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:46:04.126516 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.785284 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:46:12.785241 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:46:35.587246 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:46:35.587208 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:42.783635 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:46:42.783593 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:47:12.784572 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:47:12.784532 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:47:30.952894 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:30.952860 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.240755 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:31.240666 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:35.970219 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:35.970165 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:42.783222 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:47:42.783187 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:47:54.061094 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:54.061062 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.421964 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:54.421933 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.024383 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:58.024349 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.386555 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:47:58.386470 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.572579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:02.572544 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:03.176471 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:03.176441 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:03.534072 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:03.533976 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:03.860318 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:03.860217 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:04.419960 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.419925 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:04.423493 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423468 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26"] Oct 30 19:48:04.423727 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423712 4524 cpu_manager.go:414] "containerMap: removing container" podUID="7d7d9808-8a03-46f6-aa12-71218bde67d6" containerName="console" Oct 30 19:48:04.423772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423729 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7d9808-8a03-46f6-aa12-71218bde67d6" containerName="console" Oct 30 19:48:04.423772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423743 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3bf076c5-f276-4191-9240-8968d8b20978" containerName="insights-gathering" Oct 30 19:48:04.423772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423748 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf076c5-f276-4191-9240-8968d8b20978" containerName="insights-gathering" Oct 30 19:48:04.423772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423757 4524 cpu_manager.go:414] "containerMap: removing container" podUID="751c631e-daed-4152-8e17-f5a87972edb9" containerName="console" Oct 30 19:48:04.423772 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423762 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751c631e-daed-4152-8e17-f5a87972edb9" containerName="console" Oct 30 19:48:04.423917 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423815 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf076c5-f276-4191-9240-8968d8b20978" containerName="insights-gathering" Oct 30 19:48:04.423917 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423823 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="751c631e-daed-4152-8e17-f5a87972edb9" containerName="console" Oct 30 19:48:04.423917 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.423834 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7d9808-8a03-46f6-aa12-71218bde67d6" containerName="console" Oct 30 19:48:04.429552 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.429538 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.499276 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.499244 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kbvvk\"" Oct 30 19:48:04.548926 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.548895 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllmt\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-kube-api-access-nllmt\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.549041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.548960 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:48:04.549041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.549015 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:48:04.549041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.549024 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.549685 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.549666 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26"] Oct 30 19:48:04.649975 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.649954 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.650113 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.649999 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllmt\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-kube-api-access-nllmt\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.724545 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.724486 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllmt\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-kube-api-access-nllmt\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.727538 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.727517 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9d458d-bc5d-41ed-8639-5d1dbe3cadca-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vdw26\" (UID: \"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:04.751442 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:04.751417 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" Oct 30 19:48:05.138925 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:05.138857 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26"] Oct 30 19:48:05.141816 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:48:05.141779 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d9d458d_bc5d_41ed_8639_5d1dbe3cadca.slice/crio-ff1f0ca84a0d852b3a4c2eecea45150c3810ec7498f4d7a17d25c2feeafcb91e WatchSource:0}: Error finding container ff1f0ca84a0d852b3a4c2eecea45150c3810ec7498f4d7a17d25c2feeafcb91e: Status 404 returned error can't find the container with id ff1f0ca84a0d852b3a4c2eecea45150c3810ec7498f4d7a17d25c2feeafcb91e Oct 30 19:48:05.143839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:05.143820 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:48:05.538134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:05.538098 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" event={"ID":"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca","Type":"ContainerStarted","Data":"ff1f0ca84a0d852b3a4c2eecea45150c3810ec7498f4d7a17d25c2feeafcb91e"} Oct 30 19:48:05.742474 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:05.742436 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:09.552140 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:09.552060 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" event={"ID":"5d9d458d-bc5d-41ed-8639-5d1dbe3cadca","Type":"ContainerStarted","Data":"665b79db216e99537658eda707856b0994bc2c6d5418b726eb01e38b85a396c2"} Oct 30 19:48:09.667597 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:09.667546 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vdw26" podStartSLOduration=1.656301906 podStartE2EDuration="5.667532601s" totalImagesPullingTime="4.011230695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:09.66704793 +0000 UTC m=+384.510719561" watchObservedRunningTime="2026-10-30 19:48:09.667532601 +0000 UTC m=+384.511204230" Oct 30 19:48:12.784213 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:48:12.784179 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:48:13.156771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:13.156742 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.131419 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:14.131380 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:15.313492 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:15.313458 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:15.533472 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:15.533434 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.680510 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:21.680476 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.984094 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:21.984008 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.281202 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:22.281127 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.914396 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:23.914367 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7l7t8"] Oct 30 19:48:23.917557 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:23.917539 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:23.974113 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:23.974083 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcb2s\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-kube-api-access-qcb2s\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:23.974233 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:23.974122 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-bound-sa-token\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:23.992534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:23.992510 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7l7t8"] Oct 30 19:48:24.054440 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.054411 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gbh8c\"" Oct 30 19:48:24.074775 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.074749 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-bound-sa-token\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:24.074897 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.074836 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcb2s\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-kube-api-access-qcb2s\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:24.140348 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.140323 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-bound-sa-token\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:24.140457 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.140442 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcb2s\" (UniqueName: \"kubernetes.io/projected/5568c4b5-4d61-4479-a70e-89c21ca95791-kube-api-access-qcb2s\") pod \"cert-manager-597b95cb84-7l7t8\" (UID: \"5568c4b5-4d61-4479-a70e-89c21ca95791\") " pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:24.228315 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.228258 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7l7t8" Oct 30 19:48:24.525008 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.524920 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:24.599542 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:24.599519 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7l7t8"] Oct 30 19:48:24.602857 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:48:24.602832 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5568c4b5_4d61_4479_a70e_89c21ca95791.slice/crio-9be3ace864d3ecf9f55c7ce36975bb4390390274bb22a203b43a6e7723015620 WatchSource:0}: Error finding container 9be3ace864d3ecf9f55c7ce36975bb4390390274bb22a203b43a6e7723015620: Status 404 returned error can't find the container with id 9be3ace864d3ecf9f55c7ce36975bb4390390274bb22a203b43a6e7723015620 Oct 30 19:48:25.604821 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:25.604786 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7l7t8" event={"ID":"5568c4b5-4d61-4479-a70e-89c21ca95791","Type":"ContainerStarted","Data":"53875ff9cf57cbf6578b5ef4a138ab3a062e62e5205a2f96f3b3601a2a2f1c68"} Oct 30 19:48:25.605259 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:25.604828 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7l7t8" event={"ID":"5568c4b5-4d61-4479-a70e-89c21ca95791","Type":"ContainerStarted","Data":"9be3ace864d3ecf9f55c7ce36975bb4390390274bb22a203b43a6e7723015620"} Oct 30 19:48:25.723978 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:25.723933 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7l7t8" podStartSLOduration=2.7239211340000002 podStartE2EDuration="2.723921134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:25.72365884 +0000 UTC m=+400.567330471" watchObservedRunningTime="2026-10-30 19:48:25.723921134 +0000 UTC m=+400.567592762" Oct 30 19:48:36.392766 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:36.392614 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:36.715905 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:36.715803 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.169454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:38.169422 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.514982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:38.514904 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.812567 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:38.812491 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.242291 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.242235 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.500004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.499927 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt"] Oct 30 19:48:39.503021 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.503004 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.530011 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.529987 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.584813 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584789 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.584908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584836 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.584908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584900 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjr72\" (UniqueName: \"kubernetes.io/projected/bca56e42-576b-4d19-a433-7612128037e8-kube-api-access-cjr72\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.584990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584934 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.584990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584964 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.585072 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.584997 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.620910 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.620879 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt"] Oct 30 19:48:39.623258 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.623215 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:48:39.623368 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.623316 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:48:39.623436 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.623369 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:48:39.623490 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.623428 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:48:39.623490 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.623481 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:48:39.624863 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.624850 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vn6w\"" Oct 30 19:48:39.686254 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686233 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686284 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686426 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686408 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjr72\" (UniqueName: \"kubernetes.io/projected/bca56e42-576b-4d19-a433-7612128037e8-kube-api-access-cjr72\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686458 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686543 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686511 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686594 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686554 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.686647 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.686602 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.688869 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.688824 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca56e42-576b-4d19-a433-7612128037e8-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.689607 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.689580 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.690066 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.689680 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.690066 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.689638 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bca56e42-576b-4d19-a433-7612128037e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.750968 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.750910 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjr72\" (UniqueName: \"kubernetes.io/projected/bca56e42-576b-4d19-a433-7612128037e8-kube-api-access-cjr72\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt\" (UID: \"bca56e42-576b-4d19-a433-7612128037e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.817554 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.817533 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:39.854157 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:39.854119 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.122912 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:40.122842 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.323301 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:40.323272 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt"] Oct 30 19:48:40.334628 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:48:40.334603 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca56e42_576b_4d19_a433_7612128037e8.slice/crio-9fc6b7681da8e1184050e87a92d9064ed7cfd33fe3d97f647d6ef2862a794101 WatchSource:0}: Error finding container 9fc6b7681da8e1184050e87a92d9064ed7cfd33fe3d97f647d6ef2862a794101: Status 404 returned error can't find the container with id 9fc6b7681da8e1184050e87a92d9064ed7cfd33fe3d97f647d6ef2862a794101 Oct 30 19:48:40.650154 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:40.650116 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" event={"ID":"bca56e42-576b-4d19-a433-7612128037e8","Type":"ContainerStarted","Data":"9fc6b7681da8e1184050e87a92d9064ed7cfd33fe3d97f647d6ef2862a794101"} Oct 30 19:48:42.786108 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:48:42.786055 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:48:44.667763 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:44.667729 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" event={"ID":"bca56e42-576b-4d19-a433-7612128037e8","Type":"ContainerStarted","Data":"35785b5c3c63598c333854be79b283858235b00c748f9b3d1fc505db51c2d3aa"} Oct 30 19:48:45.672244 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:45.672204 4524 generic.go:356] "Generic (PLEG): container finished" podID="bca56e42-576b-4d19-a433-7612128037e8" containerID="35785b5c3c63598c333854be79b283858235b00c748f9b3d1fc505db51c2d3aa" exitCode=0 Oct 30 19:48:45.672688 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:45.672292 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" event={"ID":"bca56e42-576b-4d19-a433-7612128037e8","Type":"ContainerDied","Data":"35785b5c3c63598c333854be79b283858235b00c748f9b3d1fc505db51c2d3aa"} Oct 30 19:48:46.676990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:46.676957 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" event={"ID":"bca56e42-576b-4d19-a433-7612128037e8","Type":"ContainerStarted","Data":"1c7864a1f626a8bceefc9f9dd645764abc136eb00d1aac25ccec624e3507a1e3"} Oct 30 19:48:46.677467 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:46.677082 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:48:46.941820 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:46.941761 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" podStartSLOduration=2.68647692 podStartE2EDuration="7.941747459s" totalImagesPullingTime="4.775054079s" totalInitContainerRuntime="480.21646ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:46.89189255 +0000 UTC m=+421.735564224" watchObservedRunningTime="2026-10-30 19:48:46.941747459 +0000 UTC m=+421.785419141" Oct 30 19:48:47.888476 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:47.888438 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.230556 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:48.230523 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.520989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:48.520904 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:57.683818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:48:57.683789 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt" Oct 30 19:49:01.696922 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:01.696881 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.975886 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:01.975786 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.005934 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.005900 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl"] Oct 30 19:49:03.008972 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.008957 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.063524 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.063497 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:49:03.093222 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.093192 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.112867 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.112830 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:49:03.113006 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.112925 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:49:03.113006 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.112961 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4blrb\"" Oct 30 19:49:03.113006 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.112992 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:49:03.114331 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.114310 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl"] Oct 30 19:49:03.151882 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.151847 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d633935-0edc-4d93-8496-6617f6b98f41-tmp\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.152008 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.151899 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/7d633935-0edc-4d93-8496-6617f6b98f41-kube-api-access-gbt4c\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.152008 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.151926 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d633935-0edc-4d93-8496-6617f6b98f41-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.252812 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.252781 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/7d633935-0edc-4d93-8496-6617f6b98f41-kube-api-access-gbt4c\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.252812 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.252815 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d633935-0edc-4d93-8496-6617f6b98f41-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.253028 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.252873 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d633935-0edc-4d93-8496-6617f6b98f41-tmp\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.255088 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.255062 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d633935-0edc-4d93-8496-6617f6b98f41-tmp\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.255257 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.255238 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d633935-0edc-4d93-8496-6617f6b98f41-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.315810 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.315756 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/7d633935-0edc-4d93-8496-6617f6b98f41-kube-api-access-gbt4c\") pod \"kube-auth-proxy-64985d6c8c-g96kl\" (UID: \"7d633935-0edc-4d93-8496-6617f6b98f41\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.320010 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.319992 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" Oct 30 19:49:03.377387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.377355 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.692250 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.692091 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl"] Oct 30 19:49:03.694691 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:49:03.694664 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d633935_0edc_4d93_8496_6617f6b98f41.slice/crio-88dae38d53aa874fb0323a6f92fc8feef335d5b60ceab0b254ee79b0dfa6ab3e WatchSource:0}: Error finding container 88dae38d53aa874fb0323a6f92fc8feef335d5b60ceab0b254ee79b0dfa6ab3e: Status 404 returned error can't find the container with id 88dae38d53aa874fb0323a6f92fc8feef335d5b60ceab0b254ee79b0dfa6ab3e Oct 30 19:49:03.733613 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:03.733581 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" event={"ID":"7d633935-0edc-4d93-8496-6617f6b98f41","Type":"ContainerStarted","Data":"88dae38d53aa874fb0323a6f92fc8feef335d5b60ceab0b254ee79b0dfa6ab3e"} Oct 30 19:49:05.312070 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.312037 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:49:05.315017 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.315001 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.368631 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.368596 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.368817 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.368672 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.368871 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.368834 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.368924 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.368877 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.368981 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.368961 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsgq\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.369043 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.369028 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.369102 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.369073 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.369954 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.369934 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:49:05.370348 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.370327 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:49:05.370348 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.370327 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:49:05.370505 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.370335 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jf4m5\"" Oct 30 19:49:05.377402 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.377382 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:49:05.419867 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.419840 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:49:05.439487 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.439455 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:05.470104 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470075 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470251 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470128 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470251 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470160 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470251 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470188 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsgq\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470251 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470223 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470460 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470253 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470510 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.470853 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.470807 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.472837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.472812 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.472837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.472825 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.472979 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.472919 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.473016 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.472976 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.537221 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.537182 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsgq\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.537390 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.537369 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fpz9p\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.629422 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.629357 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:05.709204 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:05.709170 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.068194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:06.068158 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:49:06.071967 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:49:06.071939 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061c26de_751f_4e98_850f_68a2aa531f21.slice/crio-b162ec128f265891827dfb9a8ef76723970ac64374b165b0f0e56fb6c3410b88 WatchSource:0}: Error finding container b162ec128f265891827dfb9a8ef76723970ac64374b165b0f0e56fb6c3410b88: Status 404 returned error can't find the container with id b162ec128f265891827dfb9a8ef76723970ac64374b165b0f0e56fb6c3410b88 Oct 30 19:49:06.748441 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:06.748411 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" event={"ID":"7d633935-0edc-4d93-8496-6617f6b98f41","Type":"ContainerStarted","Data":"83c950e5909e3db7a6a5e1961e70fab6778f1b3c30921f967e47f5469e3e5df5"} Oct 30 19:49:06.749732 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:06.749711 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" event={"ID":"061c26de-751f-4e98-850f-68a2aa531f21","Type":"ContainerStarted","Data":"b162ec128f265891827dfb9a8ef76723970ac64374b165b0f0e56fb6c3410b88"} Oct 30 19:49:06.868812 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:06.868734 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-g96kl" podStartSLOduration=1.9691758620000002 podStartE2EDuration="4.8687212s" totalImagesPullingTime="2.899545338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:06.868083045 +0000 UTC m=+441.711754674" watchObservedRunningTime="2026-10-30 19:49:06.8687212 +0000 UTC m=+441.712392843" Oct 30 19:49:07.107184 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:07.107149 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:09.681026 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:09.680993 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:09.681299 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:09.681057 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:09.761058 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:09.761025 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" event={"ID":"061c26de-751f-4e98-850f-68a2aa531f21","Type":"ContainerStarted","Data":"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4"} Oct 30 19:49:09.761197 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:09.761125 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:09.911431 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:09.911375 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" podStartSLOduration=1.305298079 podStartE2EDuration="4.911360615s" totalImagesPullingTime="3.606062536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:09.910370375 +0000 UTC m=+444.754042004" watchObservedRunningTime="2026-10-30 19:49:09.911360615 +0000 UTC m=+444.755032243" Oct 30 19:49:10.766632 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:10.766588 4524 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-fpz9p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:49:10.767034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:10.766660 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" podUID="061c26de-751f-4e98-850f-68a2aa531f21" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:49:11.247653 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:11.247621 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.576978 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:11.576894 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.474903 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:12.474872 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.715083 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:12.715045 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.783242 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:49:12.783173 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:49:13.740764 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:13.740734 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.765220 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:13.765199 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:49:14.373884 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.373850 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-fjxz5"] Oct 30 19:49:14.381387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.381366 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.434801 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.434776 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 19:49:14.434956 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.434885 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-nnm8r\"" Oct 30 19:49:14.436585 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.436561 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-fjxz5"] Oct 30 19:49:14.538987 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.538959 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.538987 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.538987 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhhb\" (UniqueName: \"kubernetes.io/projected/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-kube-api-access-sqhhb\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.590713 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.590677 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:14.639960 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.639925 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.639960 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.639949 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhhb\" (UniqueName: \"kubernetes.io/projected/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-kube-api-access-sqhhb\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.640106 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:49:14.640065 4524 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 19:49:14.640143 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:49:14.640118 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs podName:57a3ea9e-fd4d-4623-81f9-91732b0c6cf9 nodeName:}" failed. No retries permitted until 2026-10-30 19:49:15.140104734 +0000 UTC m=+449.983776358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs") pod "odh-observability-584f99c788-fjxz5" (UID: "57a3ea9e-fd4d-4623-81f9-91732b0c6cf9") : secret "odh-observability-webhook-cert" not found Oct 30 19:49:14.704616 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.704587 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhhb\" (UniqueName: \"kubernetes.io/projected/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-kube-api-access-sqhhb\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:14.830163 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:14.830133 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.145337 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.145308 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:15.147799 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.147771 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/57a3ea9e-fd4d-4623-81f9-91732b0c6cf9-webhook-certs\") pod \"odh-observability-584f99c788-fjxz5\" (UID: \"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9\") " pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:15.294078 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.294042 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:15.485732 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.485641 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.664158 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.664135 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-fjxz5"] Oct 30 19:49:15.666440 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:49:15.666415 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a3ea9e_fd4d_4623_81f9_91732b0c6cf9.slice/crio-a81c36f1caaad7ec8221a7f8d0129579790448934d5ea1dae59a7736a5b238e5 WatchSource:0}: Error finding container a81c36f1caaad7ec8221a7f8d0129579790448934d5ea1dae59a7736a5b238e5: Status 404 returned error can't find the container with id a81c36f1caaad7ec8221a7f8d0129579790448934d5ea1dae59a7736a5b238e5 Oct 30 19:49:15.783955 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:15.783890 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-fjxz5" event={"ID":"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9","Type":"ContainerStarted","Data":"a81c36f1caaad7ec8221a7f8d0129579790448934d5ea1dae59a7736a5b238e5"} Oct 30 19:49:18.797285 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:18.797252 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-fjxz5" event={"ID":"57a3ea9e-fd4d-4623-81f9-91732b0c6cf9","Type":"ContainerStarted","Data":"80d8e731b2c9a86377240e6e314f5566af6249370aad1e25b628d2c469ca80b2"} Oct 30 19:49:18.797683 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:18.797396 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:18.915973 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:18.915885 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-fjxz5" podStartSLOduration=1.937178058 podStartE2EDuration="4.915871769s" totalImagesPullingTime="2.978693711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:18.914152134 +0000 UTC m=+453.757823763" watchObservedRunningTime="2026-10-30 19:49:18.915871769 +0000 UTC m=+453.759543403" Oct 30 19:49:29.803768 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:29.803737 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-fjxz5" Oct 30 19:49:32.189656 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:32.189623 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:32.726564 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:32.726530 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.664745 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:49:36.664712 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.783756 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:49:42.783717 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:50:12.784670 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:50:12.784627 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:50:17.546234 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:17.546201 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:26.758784 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.758748 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2"] Oct 30 19:50:26.760771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.760756 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.873284 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.873251 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2"] Oct 30 19:50:26.901217 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901193 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901353 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901224 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901353 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901248 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901353 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901326 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltpj\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-kube-api-access-pltpj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901371 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901399 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:26.901452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:26.901421 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa38359-9276-47a8-9f17-fb94689237e1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002714 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002666 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002721 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002754 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002772 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltpj\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-kube-api-access-pltpj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002797 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002817 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.002857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.002838 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa38359-9276-47a8-9f17-fb94689237e1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.003559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.003536 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.005119 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.005099 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.005205 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.005184 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.005461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.005442 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa38359-9276-47a8-9f17-fb94689237e1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.005500 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.005468 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8fa38359-9276-47a8-9f17-fb94689237e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.064216 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.064166 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.064522 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.064501 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltpj\" (UniqueName: \"kubernetes.io/projected/8fa38359-9276-47a8-9f17-fb94689237e1-kube-api-access-pltpj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jc4d2\" (UID: \"8fa38359-9276-47a8-9f17-fb94689237e1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.071171 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.071155 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:27.498169 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.493139 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2"] Oct 30 19:50:27.501135 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:50:27.501113 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa38359_9276_47a8_9f17_fb94689237e1.slice/crio-475e7dab133f49806c95fea0af3327d81147a3bf73d98980415ef2b6432bbde5 WatchSource:0}: Error finding container 475e7dab133f49806c95fea0af3327d81147a3bf73d98980415ef2b6432bbde5: Status 404 returned error can't find the container with id 475e7dab133f49806c95fea0af3327d81147a3bf73d98980415ef2b6432bbde5 Oct 30 19:50:27.502929 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.502900 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:27.503015 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:27.502955 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:50:28.050520 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.050481 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" event={"ID":"8fa38359-9276-47a8-9f17-fb94689237e1","Type":"ContainerStarted","Data":"2d4f42be9f88b9746b32d569d1353ee50edf0ee0239fd1ba9ff1e06438fc72a7"} Oct 30 19:50:28.050920 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.050538 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" event={"ID":"8fa38359-9276-47a8-9f17-fb94689237e1","Type":"ContainerStarted","Data":"475e7dab133f49806c95fea0af3327d81147a3bf73d98980415ef2b6432bbde5"} Oct 30 19:50:28.050920 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.050665 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:28.052309 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.052284 4524 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jc4d2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:50:28.052451 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.052325 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" podUID="8fa38359-9276-47a8-9f17-fb94689237e1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:50:28.174499 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:28.174452 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" podStartSLOduration=2.174438001 podStartE2EDuration="2.174438001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.172733327 +0000 UTC m=+523.016404956" watchObservedRunningTime="2026-10-30 19:50:28.174438001 +0000 UTC m=+523.018109630" Oct 30 19:50:29.055465 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:29.055436 4524 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jc4d2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:50:29.055849 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:29.055493 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" podUID="8fa38359-9276-47a8-9f17-fb94689237e1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:50:31.307548 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:31.307514 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.557642 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:31.557571 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.774743 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:31.774714 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.998473 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:31.998438 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.056187 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.056156 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jc4d2" Oct 30 19:50:32.239742 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.239711 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.286221 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.286153 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:50:32.286399 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.286376 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" podUID="061c26de-751f-4e98-850f-68a2aa531f21" containerName="discovery" containerID="cri-o://6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4" gracePeriod=30 Oct 30 19:50:32.493508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.493482 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.539201 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.539152 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:50:32.646827 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.646797 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.646993 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.646840 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.646993 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.646871 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.646993 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.646906 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.646993 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.646991 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.647202 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.647042 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.647202 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.647122 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\") pod \"061c26de-751f-4e98-850f-68a2aa531f21\" (UID: \"061c26de-751f-4e98-850f-68a2aa531f21\") " Oct 30 19:50:32.647428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.647392 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:32.652794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.649811 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:32.652794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.649834 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "kube-api-access-zqsgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:32.652794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.649886 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:32.652794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.650310 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:32.653188 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.653163 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:32.657954 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.657931 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert" pod "061c26de-751f-4e98-850f-68a2aa531f21" (UID: "061c26de-751f-4e98-850f-68a2aa531f21"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:32.748378 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748349 4524 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-cacerts\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748378 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748378 4524 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/061c26de-751f-4e98-850f-68a2aa531f21-local-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748391 4524 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-kubeconfig\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748403 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748416 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqsgq\" (UniqueName: \"kubernetes.io/projected/061c26de-751f-4e98-850f-68a2aa531f21-kube-api-access-zqsgq\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748431 4524 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-dns-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:32.748579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:32.748444 4524 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/061c26de-751f-4e98-850f-68a2aa531f21-istio-csr-ca-configmap\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:33.070341 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.070310 4524 generic.go:356] "Generic (PLEG): container finished" podID="061c26de-751f-4e98-850f-68a2aa531f21" containerID="6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4" exitCode=0 Oct 30 19:50:33.070513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.070392 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" Oct 30 19:50:33.070513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.070404 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" event={"ID":"061c26de-751f-4e98-850f-68a2aa531f21","Type":"ContainerDied","Data":"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4"} Oct 30 19:50:33.070513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.070481 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p" event={"ID":"061c26de-751f-4e98-850f-68a2aa531f21","Type":"ContainerDied","Data":"b162ec128f265891827dfb9a8ef76723970ac64374b165b0f0e56fb6c3410b88"} Oct 30 19:50:33.070513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.070502 4524 scope.go:122] "RemoveContainer" containerID="6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4" Oct 30 19:50:33.078561 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.078547 4524 scope.go:122] "RemoveContainer" containerID="6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4" Oct 30 19:50:33.078960 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:50:33.078939 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4\": container with ID starting with 6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4 not found: ID does not exist" containerID="6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4" Oct 30 19:50:33.079043 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.078968 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4"} err="failed to get container status \"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4\": rpc error: code = NotFound desc = could not find container \"6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4\": container with ID starting with 6f41025cb92f4f492d5d9072e370120ce044bbb396596e260770dc233ed716e4 not found: ID does not exist" Oct 30 19:50:33.234477 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.234446 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:50:33.284148 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.284118 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fpz9p"] Oct 30 19:50:33.297195 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.297170 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061c26de-751f-4e98-850f-68a2aa531f21" path="/var/lib/kubelet/pods/061c26de-751f-4e98-850f-68a2aa531f21/volumes" Oct 30 19:50:33.610849 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:33.610819 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.053136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:38.053101 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.068621 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:39.068584 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.277762 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:39.277730 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.493723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:39.493685 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.732239 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:39.732204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.939147 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:39.939112 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.169831 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:40.169798 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.459858 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:50:40.459826 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.783672 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:50:42.783643 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:51:12.784622 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:51:12.784585 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:51:35.248046 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:35.247946 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.520975 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:35.520890 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.595826 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:37.595785 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.784095 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:51:42.784060 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:51:55.684991 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:55.684945 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:57.788210 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.788177 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq"] Oct 30 19:51:57.788578 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.788451 4524 cpu_manager.go:414] "containerMap: removing container" podUID="061c26de-751f-4e98-850f-68a2aa531f21" containerName="discovery" Oct 30 19:51:57.788578 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.788461 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061c26de-751f-4e98-850f-68a2aa531f21" containerName="discovery" Oct 30 19:51:57.788578 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.788506 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="061c26de-751f-4e98-850f-68a2aa531f21" containerName="discovery" Oct 30 19:51:57.791247 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.791231 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:51:57.853641 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.853618 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-97jvh\"" Oct 30 19:51:57.903632 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.903602 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:51:57.903855 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.903799 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:51:57.904825 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.904802 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq"] Oct 30 19:51:57.954605 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:57.954579 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr927\" (UniqueName: \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\") pod \"limitador-operator-controller-manager-856dd584c5-vbmlq\" (UID: \"ebae6b6d-7291-4409-b2bf-42656b98efea\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:51:58.055371 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:58.055309 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr927\" (UniqueName: \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\") pod \"limitador-operator-controller-manager-856dd584c5-vbmlq\" (UID: \"ebae6b6d-7291-4409-b2bf-42656b98efea\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:51:58.117897 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:58.117867 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr927\" (UniqueName: \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\") pod \"limitador-operator-controller-manager-856dd584c5-vbmlq\" (UID: \"ebae6b6d-7291-4409-b2bf-42656b98efea\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:51:58.402716 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:58.402600 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:51:58.563142 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:58.563105 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:58.773632 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:58.773564 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq"] Oct 30 19:51:58.776791 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:51:58.776763 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebae6b6d_7291_4409_b2bf_42656b98efea.slice/crio-f58b755908d4c4000c907b7986cdf070a1cba72a1a13f77749db594f93eb9c87 WatchSource:0}: Error finding container f58b755908d4c4000c907b7986cdf070a1cba72a1a13f77749db594f93eb9c87: Status 404 returned error can't find the container with id f58b755908d4c4000c907b7986cdf070a1cba72a1a13f77749db594f93eb9c87 Oct 30 19:51:59.373607 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.373571 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" event={"ID":"ebae6b6d-7291-4409-b2bf-42656b98efea","Type":"ContainerStarted","Data":"f58b755908d4c4000c907b7986cdf070a1cba72a1a13f77749db594f93eb9c87"} Oct 30 19:51:59.814458 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.814421 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8"] Oct 30 19:51:59.817410 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.817394 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:51:59.873034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.873003 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:51:59.921780 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.921754 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l2v8w\"" Oct 30 19:51:59.936978 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.936949 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8"] Oct 30 19:51:59.972395 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:51:59.972364 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swpq\" (UniqueName: \"kubernetes.io/projected/79bcaf70-b3ba-4651-941b-4e2f6e6e93eb-kube-api-access-5swpq\") pod \"dns-operator-controller-manager-65bfd79598-bkrk8\" (UID: \"79bcaf70-b3ba-4651-941b-4e2f6e6e93eb\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:00.073786 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.073692 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swpq\" (UniqueName: \"kubernetes.io/projected/79bcaf70-b3ba-4651-941b-4e2f6e6e93eb-kube-api-access-5swpq\") pod \"dns-operator-controller-manager-65bfd79598-bkrk8\" (UID: \"79bcaf70-b3ba-4651-941b-4e2f6e6e93eb\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:00.139722 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.139675 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swpq\" (UniqueName: \"kubernetes.io/projected/79bcaf70-b3ba-4651-941b-4e2f6e6e93eb-kube-api-access-5swpq\") pod \"dns-operator-controller-manager-65bfd79598-bkrk8\" (UID: \"79bcaf70-b3ba-4651-941b-4e2f6e6e93eb\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:00.430204 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.430178 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:00.436074 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.436048 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:00.661836 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.661803 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:00.845575 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:00.845511 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8"] Oct 30 19:52:00.849811 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:00.849783 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bcaf70_b3ba_4651_941b_4e2f6e6e93eb.slice/crio-ebf5406f09b993fe122dce9fe881fe83b818eca4bd63dc0f1be68b0344d426d6 WatchSource:0}: Error finding container ebf5406f09b993fe122dce9fe881fe83b818eca4bd63dc0f1be68b0344d426d6: Status 404 returned error can't find the container with id ebf5406f09b993fe122dce9fe881fe83b818eca4bd63dc0f1be68b0344d426d6 Oct 30 19:52:01.174450 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:01.174416 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.383714 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:01.383663 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" event={"ID":"79bcaf70-b3ba-4651-941b-4e2f6e6e93eb","Type":"ContainerStarted","Data":"ebf5406f09b993fe122dce9fe881fe83b818eca4bd63dc0f1be68b0344d426d6"} Oct 30 19:52:01.691608 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:01.691570 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.012196 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:02.012104 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.677104 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:02.677056 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.926592 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:02.926556 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.396684 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:03.396645 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" event={"ID":"ebae6b6d-7291-4409-b2bf-42656b98efea","Type":"ContainerStarted","Data":"ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed"} Oct 30 19:52:03.396874 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:03.396775 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:52:03.543121 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:03.543056 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" podStartSLOduration=2.774569873 podStartE2EDuration="6.543037969s" totalImagesPullingTime="3.768468096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:03.542287112 +0000 UTC m=+618.385958926" watchObservedRunningTime="2026-10-30 19:52:03.543037969 +0000 UTC m=+618.386709598" Oct 30 19:52:04.402097 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.402071 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" event={"ID":"79bcaf70-b3ba-4651-941b-4e2f6e6e93eb","Type":"ContainerStarted","Data":"dd05ffe4abb85fd083f0036085311e867ed0920264e4b140b8e65bfa4886a8d0"} Oct 30 19:52:04.402452 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.402357 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:04.580468 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.580380 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" podStartSLOduration=2.141947108 podStartE2EDuration="5.580366668s" totalImagesPullingTime="3.43841956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:04.579370876 +0000 UTC m=+619.423042506" watchObservedRunningTime="2026-10-30 19:52:04.580366668 +0000 UTC m=+619.424038297" Oct 30 19:52:04.752659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.752623 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm"] Oct 30 19:52:04.755879 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.755857 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:04.766228 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.766204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:04.813261 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.813234 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-78rg8\"" Oct 30 19:52:04.848166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.848099 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm"] Oct 30 19:52:04.919519 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.919494 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74jf\" (UniqueName: \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:04.919665 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:04.919527 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.020744 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.020711 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74jf\" (UniqueName: \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.020892 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.020752 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.021145 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.021126 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.088977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.088946 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74jf\" (UniqueName: \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.199845 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.199810 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.368383 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.368340 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:05.843915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:05.843888 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm"] Oct 30 19:52:05.847432 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:05.847403 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49507ffe_e3d9_43ff_9b50_9a65f4be1f5b.slice/crio-ff7959413084b8a4f764811af9a0cbdb3be4c8b9e5824c608b4236e74855e322 WatchSource:0}: Error finding container ff7959413084b8a4f764811af9a0cbdb3be4c8b9e5824c608b4236e74855e322: Status 404 returned error can't find the container with id ff7959413084b8a4f764811af9a0cbdb3be4c8b9e5824c608b4236e74855e322 Oct 30 19:52:06.413037 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:06.412994 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" event={"ID":"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b","Type":"ContainerStarted","Data":"ff7959413084b8a4f764811af9a0cbdb3be4c8b9e5824c608b4236e74855e322"} Oct 30 19:52:06.709340 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:06.709249 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:06.954463 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:06.954425 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:08.558694 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:08.558648 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:08.775019 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:08.774986 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.584770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:10.584739 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.824227 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:10.824198 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.790922 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:12.790896 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:52:13.444096 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:13.444062 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" event={"ID":"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b","Type":"ContainerStarted","Data":"d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e"} Oct 30 19:52:13.444263 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:13.444172 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:13.674579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:13.674519 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" podStartSLOduration=2.799478468 podStartE2EDuration="9.674502356s" totalImagesPullingTime="6.875023888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:13.623824123 +0000 UTC m=+628.467495754" watchObservedRunningTime="2026-10-30 19:52:13.674502356 +0000 UTC m=+628.518173985" Oct 30 19:52:14.404420 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:14.404386 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:52:15.410537 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:15.410508 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-bkrk8" Oct 30 19:52:16.544983 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:16.544955 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.798574 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:16.798497 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:18.554179 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:18.554146 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:18.816148 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:18.816075 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.966278 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:20.966246 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.195639 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:21.195610 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:24.450873 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:24.450842 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:25.476072 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:25.476038 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.792422 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:25.792343 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:27.245244 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.245215 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl"] Oct 30 19:52:27.250005 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.249989 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.278527 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.278506 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl"] Oct 30 19:52:27.284586 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.284565 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.284678 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.284611 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmt26\" (UniqueName: \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.385785 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.385760 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.385885 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.385810 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmt26\" (UniqueName: \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.386140 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.386121 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.450668 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.450638 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmt26\" (UniqueName: \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\") pod \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.561324 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.561270 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:27.995536 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.995020 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:27.996131 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:27.996098 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl"] Oct 30 19:52:27.997114 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:27.997082 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd55448_c3ec_4a70_b00e_bf69a4f7e130.slice/crio-47a4e379ac110183f5e31e0aed24953ad260326d4ab66a52efe5ef9e07265ba6 WatchSource:0}: Error finding container 47a4e379ac110183f5e31e0aed24953ad260326d4ab66a52efe5ef9e07265ba6: Status 404 returned error can't find the container with id 47a4e379ac110183f5e31e0aed24953ad260326d4ab66a52efe5ef9e07265ba6 Oct 30 19:52:28.241423 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.241387 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:28.504760 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.504727 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" event={"ID":"2cd55448-c3ec-4a70-b00e-bf69a4f7e130","Type":"ContainerStarted","Data":"1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8"} Oct 30 19:52:28.505133 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.504784 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" event={"ID":"2cd55448-c3ec-4a70-b00e-bf69a4f7e130","Type":"ContainerStarted","Data":"47a4e379ac110183f5e31e0aed24953ad260326d4ab66a52efe5ef9e07265ba6"} Oct 30 19:52:28.505133 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.504847 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:28.688651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.688605 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" podStartSLOduration=1.688590576 podStartE2EDuration="1.688590576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:28.687207073 +0000 UTC m=+643.530878705" watchObservedRunningTime="2026-10-30 19:52:28.688590576 +0000 UTC m=+643.532262205" Oct 30 19:52:28.807978 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.807900 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm"] Oct 30 19:52:28.808398 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.808202 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" containerName="manager" containerID="cri-o://d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e" gracePeriod=2 Oct 30 19:52:28.811101 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.811076 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm"] Oct 30 19:52:28.844299 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.844277 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl"] Oct 30 19:52:28.857549 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.857521 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:28.857922 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.857899 4524 cpu_manager.go:414] "containerMap: removing container" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" containerName="manager" Oct 30 19:52:28.857922 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.857919 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" containerName="manager" Oct 30 19:52:28.858091 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.858059 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" containerName="manager" Oct 30 19:52:28.860213 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:28.860188 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:28.861166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.861148 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl"] Oct 30 19:52:28.861371 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.861349 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.884024 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.883996 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:28.884445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.884426 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" containerName="manager" Oct 30 19:52:28.884445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.884445 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" containerName="manager" Oct 30 19:52:28.884562 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.884533 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" containerName="manager" Oct 30 19:52:28.887440 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.887416 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq"] Oct 30 19:52:28.887541 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.887522 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:28.887792 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.887766 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" containerName="manager" containerID="cri-o://ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed" gracePeriod=2 Oct 30 19:52:28.897287 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.897261 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk5l\" (UniqueName: \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.897414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.897304 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.897414 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.897385 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:28.897526 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.897472 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9rl\" (UniqueName: \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:28.908625 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.908602 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq"] Oct 30 19:52:28.931284 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.931259 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk"] Oct 30 19:52:28.931569 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.931557 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" containerName="manager" Oct 30 19:52:28.931616 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.931571 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" containerName="manager" Oct 30 19:52:28.931655 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.931620 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" containerName="manager" Oct 30 19:52:28.934326 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.934301 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:28.969844 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.969822 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:28.982175 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.982154 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:28.998865 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.998841 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9rl\" (UniqueName: \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:28.998994 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.998890 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk5l\" (UniqueName: \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.998994 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.998910 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.998994 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.998927 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:28.998994 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.998963 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f22\" (UniqueName: \"kubernetes.io/projected/db72c297-f924-45ba-aa81-81a0f9945c13-kube-api-access-h9f22\") pod \"limitador-operator-controller-manager-856dd584c5-rs8pk\" (UID: \"db72c297-f924-45ba-aa81-81a0f9945c13\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:28.999305 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.999276 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:28.999416 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:28.999355 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:29.006405 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.006374 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk"] Oct 30 19:52:29.047142 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:29.046994 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:29.067579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.067513 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9rl\" (UniqueName: \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\") pod \"kuadrant-operator-controller-manager-956956bc4-2rc8h\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:29.067721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.067607 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk5l\" (UniqueName: \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-vl8jz\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:29.099383 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.099355 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f22\" (UniqueName: \"kubernetes.io/projected/db72c297-f924-45ba-aa81-81a0f9945c13-kube-api-access-h9f22\") pod \"limitador-operator-controller-manager-856dd584c5-rs8pk\" (UID: \"db72c297-f924-45ba-aa81-81a0f9945c13\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:29.100842 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:29.100799 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-jvflm\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:29.163051 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.163022 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f22\" (UniqueName: \"kubernetes.io/projected/db72c297-f924-45ba-aa81-81a0f9945c13-kube-api-access-h9f22\") pod \"limitador-operator-controller-manager-856dd584c5-rs8pk\" (UID: \"db72c297-f924-45ba-aa81-81a0f9945c13\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:29.178533 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.178513 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:29.180494 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.180477 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:52:29.199217 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.199190 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:29.200341 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.200321 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\" (UniqueName: \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\") pod \"ebae6b6d-7291-4409-b2bf-42656b98efea\" (UID: \"ebae6b6d-7291-4409-b2bf-42656b98efea\") " Oct 30 19:52:29.200399 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.200383 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\") pod \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " Oct 30 19:52:29.200468 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.200454 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\" (UniqueName: \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\") pod \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\" (UID: \"49507ffe-e3d9-43ff-9b50-9a65f4be1f5b\") " Oct 30 19:52:29.200889 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.200867 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume" pod "49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" (UID: "49507ffe-e3d9-43ff-9b50-9a65f4be1f5b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:29.202390 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.202362 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf" pod "49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" (UID: "49507ffe-e3d9-43ff-9b50-9a65f4be1f5b"). InnerVolumeSpecName "kube-api-access-j74jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:29.202461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.202409 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927" pod "ebae6b6d-7291-4409-b2bf-42656b98efea" (UID: "ebae6b6d-7291-4409-b2bf-42656b98efea"). InnerVolumeSpecName "kube-api-access-jr927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:29.250523 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.250496 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:29.255691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.255649 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:29.272537 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:29.272504 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-vbmlq\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:52:29.298060 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.298021 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" path="/var/lib/kubelet/pods/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b/volumes" Oct 30 19:52:29.298352 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.298340 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebae6b6d-7291-4409-b2bf-42656b98efea" path="/var/lib/kubelet/pods/ebae6b6d-7291-4409-b2bf-42656b98efea/volumes" Oct 30 19:52:29.301230 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.301137 4524 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-extensions-socket-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:29.301230 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.301165 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j74jf\" (UniqueName: \"kubernetes.io/projected/49507ffe-e3d9-43ff-9b50-9a65f4be1f5b-kube-api-access-j74jf\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:29.301230 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.301182 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr927\" (UniqueName: \"kubernetes.io/projected/ebae6b6d-7291-4409-b2bf-42656b98efea-kube-api-access-jr927\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:29.510401 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.510367 4524 generic.go:356] "Generic (PLEG): container finished" podID="49507ffe-e3d9-43ff-9b50-9a65f4be1f5b" containerID="d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e" exitCode=0 Oct 30 19:52:29.510864 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.510411 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jvflm" Oct 30 19:52:29.510864 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.510477 4524 scope.go:122] "RemoveContainer" containerID="d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e" Oct 30 19:52:29.511908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.511884 4524 generic.go:356] "Generic (PLEG): container finished" podID="ebae6b6d-7291-4409-b2bf-42656b98efea" containerID="ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed" exitCode=0 Oct 30 19:52:29.512029 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.511935 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vbmlq" Oct 30 19:52:29.518996 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.518979 4524 scope.go:122] "RemoveContainer" containerID="d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e" Oct 30 19:52:29.519273 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:29.519256 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e\": container with ID starting with d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e not found: ID does not exist" containerID="d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e" Oct 30 19:52:29.519329 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.519283 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e"} err="failed to get container status \"d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e\": rpc error: code = NotFound desc = could not find container \"d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e\": container with ID starting with d277e87c40bc62b5ca454da868025e7a619c9c9f0b85ad4df0a96d9aa69d442e not found: ID does not exist" Oct 30 19:52:29.519329 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.519301 4524 scope.go:122] "RemoveContainer" containerID="ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed" Oct 30 19:52:29.526345 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.526329 4524 scope.go:122] "RemoveContainer" containerID="ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed" Oct 30 19:52:29.526621 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:29.526602 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed\": container with ID starting with ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed not found: ID does not exist" containerID="ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed" Oct 30 19:52:29.526730 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.526623 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed"} err="failed to get container status \"ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed\": rpc error: code = NotFound desc = could not find container \"ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed\": container with ID starting with ea8d4c9074acd67a59d37eb99ef822d9cc1a9bfa2c9570b8529ed79f89df12ed not found: ID does not exist" Oct 30 19:52:29.574445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.574425 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:29.577280 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:29.577256 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cfd808_b5c2_409d_95a1_0c36935ddd71.slice/crio-b801dcbed29abfacabb6be1ea110330035be92841cd5533f40dce854ecbd3b6c WatchSource:0}: Error finding container b801dcbed29abfacabb6be1ea110330035be92841cd5533f40dce854ecbd3b6c: Status 404 returned error can't find the container with id b801dcbed29abfacabb6be1ea110330035be92841cd5533f40dce854ecbd3b6c Oct 30 19:52:29.645265 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.645236 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk"] Oct 30 19:52:29.647851 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:29.647829 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb72c297_f924_45ba_aa81_81a0f9945c13.slice/crio-dc6f66f431e4dafb0219c40d41bfff5d03e2ec1e36be8b58fdaa638bfa2ca9f3 WatchSource:0}: Error finding container dc6f66f431e4dafb0219c40d41bfff5d03e2ec1e36be8b58fdaa638bfa2ca9f3: Status 404 returned error can't find the container with id dc6f66f431e4dafb0219c40d41bfff5d03e2ec1e36be8b58fdaa638bfa2ca9f3 Oct 30 19:52:29.653913 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:29.653892 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:29.658522 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:29.658502 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf02e1e0_9e40_4a9a_9c39_fb482a4b236e.slice/crio-2a48b189e70dada301965682c0b8be12010066daa8a091edae8052608791f85b WatchSource:0}: Error finding container 2a48b189e70dada301965682c0b8be12010066daa8a091edae8052608791f85b: Status 404 returned error can't find the container with id 2a48b189e70dada301965682c0b8be12010066daa8a091edae8052608791f85b Oct 30 19:52:30.521390 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.521353 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" event={"ID":"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e","Type":"ContainerStarted","Data":"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05"} Oct 30 19:52:30.521833 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.521451 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" event={"ID":"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e","Type":"ContainerStarted","Data":"2a48b189e70dada301965682c0b8be12010066daa8a091edae8052608791f85b"} Oct 30 19:52:30.521833 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.521484 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:30.524756 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.524724 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" event={"ID":"a9cfd808-b5c2-409d-95a1-0c36935ddd71","Type":"ContainerStarted","Data":"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671"} Oct 30 19:52:30.524914 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.524770 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" event={"ID":"a9cfd808-b5c2-409d-95a1-0c36935ddd71","Type":"ContainerStarted","Data":"b801dcbed29abfacabb6be1ea110330035be92841cd5533f40dce854ecbd3b6c"} Oct 30 19:52:30.524914 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.524803 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:30.526256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.526231 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" event={"ID":"db72c297-f924-45ba-aa81-81a0f9945c13","Type":"ContainerStarted","Data":"2c55eef21b92597be56bf7783f64a4724b454ab6533ee274cfaf58df43e9de8c"} Oct 30 19:52:30.526393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.526279 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:30.526393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.526289 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" containerName="manager" containerID="cri-o://1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8" gracePeriod=2 Oct 30 19:52:30.526393 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.526296 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" event={"ID":"db72c297-f924-45ba-aa81-81a0f9945c13","Type":"ContainerStarted","Data":"dc6f66f431e4dafb0219c40d41bfff5d03e2ec1e36be8b58fdaa638bfa2ca9f3"} Oct 30 19:52:30.658255 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.658209 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" podStartSLOduration=2.65819569 podStartE2EDuration="2.65819569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:30.657248159 +0000 UTC m=+645.500919792" watchObservedRunningTime="2026-10-30 19:52:30.65819569 +0000 UTC m=+645.501867319" Oct 30 19:52:30.748293 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.748262 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:30.794839 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.794681 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" podStartSLOduration=2.794662065 podStartE2EDuration="2.794662065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:30.793519408 +0000 UTC m=+645.637191037" watchObservedRunningTime="2026-10-30 19:52:30.794662065 +0000 UTC m=+645.638333695" Oct 30 19:52:30.853598 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.853573 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:30.912027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.912000 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\" (UniqueName: \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\") pod \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " Oct 30 19:52:30.912156 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.912039 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\") pod \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\" (UID: \"2cd55448-c3ec-4a70-b00e-bf69a4f7e130\") " Oct 30 19:52:30.912479 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.912458 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume" pod "2cd55448-c3ec-4a70-b00e-bf69a4f7e130" (UID: "2cd55448-c3ec-4a70-b00e-bf69a4f7e130"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:30.914585 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.914304 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" podStartSLOduration=2.914287942 podStartE2EDuration="2.914287942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:30.913426023 +0000 UTC m=+645.757097653" watchObservedRunningTime="2026-10-30 19:52:30.914287942 +0000 UTC m=+645.757959566" Oct 30 19:52:30.914585 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:30.914369 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26" pod "2cd55448-c3ec-4a70-b00e-bf69a4f7e130" (UID: "2cd55448-c3ec-4a70-b00e-bf69a4f7e130"). InnerVolumeSpecName "kube-api-access-gmt26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:30.969427 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:30.969399 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-dvjxl\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:31.012916 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.012892 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmt26\" (UniqueName: \"kubernetes.io/projected/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-kube-api-access-gmt26\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:31.012916 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.012911 4524 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cd55448-c3ec-4a70-b00e-bf69a4f7e130-extensions-socket-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:31.047295 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.047237 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:31.297515 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.297444 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" path="/var/lib/kubelet/pods/2cd55448-c3ec-4a70-b00e-bf69a4f7e130/volumes" Oct 30 19:52:31.531745 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.531691 4524 generic.go:356] "Generic (PLEG): container finished" podID="2cd55448-c3ec-4a70-b00e-bf69a4f7e130" containerID="1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8" exitCode=0 Oct 30 19:52:31.532181 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.531754 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-dvjxl" Oct 30 19:52:31.532181 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.531803 4524 scope.go:122] "RemoveContainer" containerID="1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8" Oct 30 19:52:31.539644 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.539628 4524 scope.go:122] "RemoveContainer" containerID="1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8" Oct 30 19:52:31.539918 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:31.539901 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8\": container with ID starting with 1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8 not found: ID does not exist" containerID="1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8" Oct 30 19:52:31.539969 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:31.539928 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8"} err="failed to get container status \"1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8\": rpc error: code = NotFound desc = could not find container \"1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8\": container with ID starting with 1fa4a949c7dd2226fe256d810e321d9c999f7d8f5c26650fe46592634a0b5ea8 not found: ID does not exist" Oct 30 19:52:35.744254 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:35.744213 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.024588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:36.024493 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:37.715019 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:37.714985 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.534367 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:41.534341 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:41.534771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:41.534416 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:41.534771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:41.534571 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rs8pk" Oct 30 19:52:41.985652 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:41.985621 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:41.985898 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:41.985872 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" podUID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" containerName="manager" containerID="cri-o://ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671" gracePeriod=10 Oct 30 19:52:42.235795 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.235735 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:42.297807 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.297780 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\" (UniqueName: \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\") pod \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " Oct 30 19:52:42.297926 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.297884 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\") pod \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\" (UID: \"a9cfd808-b5c2-409d-95a1-0c36935ddd71\") " Oct 30 19:52:42.298248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.298222 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume" pod "a9cfd808-b5c2-409d-95a1-0c36935ddd71" (UID: "a9cfd808-b5c2-409d-95a1-0c36935ddd71"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:42.299669 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.299648 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l" pod "a9cfd808-b5c2-409d-95a1-0c36935ddd71" (UID: "a9cfd808-b5c2-409d-95a1-0c36935ddd71"). InnerVolumeSpecName "kube-api-access-brk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:42.398976 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.398943 4524 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9cfd808-b5c2-409d-95a1-0c36935ddd71-extensions-socket-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:42.398976 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.398972 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brk5l\" (UniqueName: \"kubernetes.io/projected/a9cfd808-b5c2-409d-95a1-0c36935ddd71-kube-api-access-brk5l\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:42.450171 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.450139 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 19:52:42.450447 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.450435 4524 cpu_manager.go:414] "containerMap: removing container" podUID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" containerName="manager" Oct 30 19:52:42.450497 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.450449 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" containerName="manager" Oct 30 19:52:42.450534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.450505 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" containerName="manager" Oct 30 19:52:42.452276 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.452260 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.499977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.499921 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.500080 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.500028 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jsv\" (UniqueName: \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.518296 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.518265 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 19:52:42.577665 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.577623 4524 generic.go:356] "Generic (PLEG): container finished" podID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" containerID="ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671" exitCode=0 Oct 30 19:52:42.578004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.577669 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" Oct 30 19:52:42.578004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.577722 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" event={"ID":"a9cfd808-b5c2-409d-95a1-0c36935ddd71","Type":"ContainerDied","Data":"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671"} Oct 30 19:52:42.578004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.577782 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz" event={"ID":"a9cfd808-b5c2-409d-95a1-0c36935ddd71","Type":"ContainerDied","Data":"b801dcbed29abfacabb6be1ea110330035be92841cd5533f40dce854ecbd3b6c"} Oct 30 19:52:42.578004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.577809 4524 scope.go:122] "RemoveContainer" containerID="ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671" Oct 30 19:52:42.586164 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.586148 4524 scope.go:122] "RemoveContainer" containerID="ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671" Oct 30 19:52:42.586410 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:42.586392 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671\": container with ID starting with ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671 not found: ID does not exist" containerID="ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671" Oct 30 19:52:42.586454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.586417 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671"} err="failed to get container status \"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671\": rpc error: code = NotFound desc = could not find container \"ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671\": container with ID starting with ce0f2e80af808e2e4ea3806d8e9f1c97c6f7a6c366e2c577dd30dc4c296a9671 not found: ID does not exist" Oct 30 19:52:42.600833 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.600815 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jsv\" (UniqueName: \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.600909 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.600860 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.601167 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.601152 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.670101 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.670076 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jsv\" (UniqueName: \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5mqln\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.751646 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.751587 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:42.759738 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.759717 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-vl8jz"] Oct 30 19:52:42.763236 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:42.763220 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:42.784465 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:42.784413 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:52:43.191943 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.191479 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 19:52:43.194761 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:52:43.194691 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86f8f1d_e792_4041_b3f9_69cdeb26932c.slice/crio-51b4090a78d828e0183bd686edcb58c5e950b8ecbad4884fac43a28018827c6c WatchSource:0}: Error finding container 51b4090a78d828e0183bd686edcb58c5e950b8ecbad4884fac43a28018827c6c: Status 404 returned error can't find the container with id 51b4090a78d828e0183bd686edcb58c5e950b8ecbad4884fac43a28018827c6c Oct 30 19:52:43.298501 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.298470 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cfd808-b5c2-409d-95a1-0c36935ddd71" path="/var/lib/kubelet/pods/a9cfd808-b5c2-409d-95a1-0c36935ddd71/volumes" Oct 30 19:52:43.584672 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.584594 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" event={"ID":"d86f8f1d-e792-4041-b3f9-69cdeb26932c","Type":"ContainerStarted","Data":"ad00ae6b99675f6442d133da195e58eb128d15005281723ac7e633226db2ceae"} Oct 30 19:52:43.584672 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.584641 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" event={"ID":"d86f8f1d-e792-4041-b3f9-69cdeb26932c","Type":"ContainerStarted","Data":"51b4090a78d828e0183bd686edcb58c5e950b8ecbad4884fac43a28018827c6c"} Oct 30 19:52:43.585069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.584742 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:43.711587 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:43.711532 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" podStartSLOduration=1.711516838 podStartE2EDuration="1.711516838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:43.71109064 +0000 UTC m=+658.554762272" watchObservedRunningTime="2026-10-30 19:52:43.711516838 +0000 UTC m=+658.555188468" Oct 30 19:52:44.448505 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:44.448478 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:44.734662 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:44.734565 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.239411 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:46.239370 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.591485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:54.591448 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 19:52:54.814254 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:54.814225 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:54.814450 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:54.814426 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" podUID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" containerName="manager" containerID="cri-o://0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05" gracePeriod=10 Oct 30 19:52:55.054963 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.054938 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:55.199239 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.199168 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\" (UniqueName: \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\") pod \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " Oct 30 19:52:55.199374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.199238 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\") pod \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\" (UID: \"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e\") " Oct 30 19:52:55.199589 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.199559 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume" pod "bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" (UID: "bf02e1e0-9e40-4a9a-9c39-fb482a4b236e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:55.201139 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.201108 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl" pod "bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" (UID: "bf02e1e0-9e40-4a9a-9c39-fb482a4b236e"). InnerVolumeSpecName "kube-api-access-zg9rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:55.299739 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.299688 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9rl\" (UniqueName: \"kubernetes.io/projected/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-kube-api-access-zg9rl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:55.299739 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.299732 4524 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e-extensions-socket-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:55.630818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.630738 4524 generic.go:356] "Generic (PLEG): container finished" podID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" containerID="0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05" exitCode=0 Oct 30 19:52:55.630818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.630785 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" Oct 30 19:52:55.630818 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.630800 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" event={"ID":"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e","Type":"ContainerDied","Data":"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05"} Oct 30 19:52:55.631310 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.630850 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h" event={"ID":"bf02e1e0-9e40-4a9a-9c39-fb482a4b236e","Type":"ContainerDied","Data":"2a48b189e70dada301965682c0b8be12010066daa8a091edae8052608791f85b"} Oct 30 19:52:55.631310 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.630880 4524 scope.go:122] "RemoveContainer" containerID="0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05" Oct 30 19:52:55.638855 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.638835 4524 scope.go:122] "RemoveContainer" containerID="0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05" Oct 30 19:52:55.639122 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:52:55.639093 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05\": container with ID starting with 0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05 not found: ID does not exist" containerID="0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05" Oct 30 19:52:55.639164 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.639132 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05"} err="failed to get container status \"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05\": rpc error: code = NotFound desc = could not find container \"0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05\": container with ID starting with 0e010e97f4ecd42fdd48ba82cfdf5560cb17c84a080f5a3a05e58a01729f6d05 not found: ID does not exist" Oct 30 19:52:55.809770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.809744 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:55.821786 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:55.821762 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-2rc8h"] Oct 30 19:52:57.297773 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:57.297680 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" path="/var/lib/kubelet/pods/bf02e1e0-9e40-4a9a-9c39-fb482a4b236e/volumes" Oct 30 19:52:59.905951 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:52:59.905906 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.164356 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:00.164263 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.783177 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:53:12.783140 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:53:16.351182 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.351140 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.365780 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.365750 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:16.366124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.366107 4524 cpu_manager.go:414] "containerMap: removing container" podUID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" containerName="manager" Oct 30 19:53:16.366205 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.366125 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" containerName="manager" Oct 30 19:53:16.366257 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.366216 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf02e1e0-9e40-4a9a-9c39-fb482a4b236e" containerName="manager" Oct 30 19:53:16.368136 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.368116 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.420843 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.420820 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:53:16.421178 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.421162 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s9c2h\"" Oct 30 19:53:16.466499 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.466474 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.466601 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.466527 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsfh\" (UniqueName: \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.472134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.472108 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:16.567438 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.567418 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.567534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.567479 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsfh\" (UniqueName: \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.568095 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.568077 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.611819 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.611754 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:16.628761 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.628731 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsfh\" (UniqueName: \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\") pod \"limitador-limitador-74dfcfb894-tx4jh\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.678878 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.678853 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:16.723972 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.723935 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.933247 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:16.933212 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.051391 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:17.051364 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:17.054606 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:53:17.054579 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7edd38_7db9_4252_9996_7538308e3379.slice/crio-f69c593469ded51c4fbdd36b5cc24f159c44c73ad5b3f69473fd79802df42517 WatchSource:0}: Error finding container f69c593469ded51c4fbdd36b5cc24f159c44c73ad5b3f69473fd79802df42517: Status 404 returned error can't find the container with id f69c593469ded51c4fbdd36b5cc24f159c44c73ad5b3f69473fd79802df42517 Oct 30 19:53:17.056634 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:17.056614 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:53:17.720274 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:17.720238 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" event={"ID":"4d7edd38-7db9-4252-9996-7538308e3379","Type":"ContainerStarted","Data":"f69c593469ded51c4fbdd36b5cc24f159c44c73ad5b3f69473fd79802df42517"} Oct 30 19:53:18.083771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:18.083644 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:18.298582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:18.298540 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:20.734276 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:20.734237 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" event={"ID":"4d7edd38-7db9-4252-9996-7538308e3379","Type":"ContainerStarted","Data":"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40"} Oct 30 19:53:20.734650 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:20.734318 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:20.850830 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:20.850666 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" podStartSLOduration=2.225893754 podStartE2EDuration="4.850649261s" totalImagesPullingTime="2.624755507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:20.850121759 +0000 UTC m=+695.693793388" watchObservedRunningTime="2026-10-30 19:53:20.850649261 +0000 UTC m=+695.694320893" Oct 30 19:53:26.055021 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:26.054986 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.740008 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:31.739981 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:35.169485 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.169450 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:35.169921 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.169676 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" podUID="4d7edd38-7db9-4252-9996-7538308e3379" containerName="limitador" containerID="cri-o://676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40" gracePeriod=30 Oct 30 19:53:35.703075 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.703051 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:35.802721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.802647 4524 generic.go:356] "Generic (PLEG): container finished" podID="4d7edd38-7db9-4252-9996-7538308e3379" containerID="676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40" exitCode=0 Oct 30 19:53:35.802721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.802686 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" Oct 30 19:53:35.802868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.802736 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" event={"ID":"4d7edd38-7db9-4252-9996-7538308e3379","Type":"ContainerDied","Data":"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40"} Oct 30 19:53:35.802868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.802796 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tx4jh" event={"ID":"4d7edd38-7db9-4252-9996-7538308e3379","Type":"ContainerDied","Data":"f69c593469ded51c4fbdd36b5cc24f159c44c73ad5b3f69473fd79802df42517"} Oct 30 19:53:35.802868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.802824 4524 scope.go:122] "RemoveContainer" containerID="676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40" Oct 30 19:53:35.808576 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.808560 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\" (UniqueName: \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\") pod \"4d7edd38-7db9-4252-9996-7538308e3379\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " Oct 30 19:53:35.808674 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.808614 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\" (UniqueName: \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\") pod \"4d7edd38-7db9-4252-9996-7538308e3379\" (UID: \"4d7edd38-7db9-4252-9996-7538308e3379\") " Oct 30 19:53:35.809017 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.808987 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file" pod "4d7edd38-7db9-4252-9996-7538308e3379" (UID: "4d7edd38-7db9-4252-9996-7538308e3379"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:35.810479 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.810460 4524 scope.go:122] "RemoveContainer" containerID="676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40" Oct 30 19:53:35.810559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.810501 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh" pod "4d7edd38-7db9-4252-9996-7538308e3379" (UID: "4d7edd38-7db9-4252-9996-7538308e3379"). InnerVolumeSpecName "kube-api-access-2xsfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:35.810755 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:53:35.810737 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40\": container with ID starting with 676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40 not found: ID does not exist" containerID="676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40" Oct 30 19:53:35.810838 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.810766 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40"} err="failed to get container status \"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40\": rpc error: code = NotFound desc = could not find container \"676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40\": container with ID starting with 676ae532af00a19b39436e076623cddc7fac3c9d5e66cea8d6339e3651716a40 not found: ID does not exist" Oct 30 19:53:35.909508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.909484 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsfh\" (UniqueName: \"kubernetes.io/projected/4d7edd38-7db9-4252-9996-7538308e3379-kube-api-access-2xsfh\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:35.909508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:35.909506 4524 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d7edd38-7db9-4252-9996-7538308e3379-config-file\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:36.223183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:36.223151 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:36.272977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:36.272948 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tx4jh"] Oct 30 19:53:37.302877 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:37.302843 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7edd38-7db9-4252-9996-7538308e3379" path="/var/lib/kubelet/pods/4d7edd38-7db9-4252-9996-7538308e3379/volumes" Oct 30 19:53:38.054814 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:38.054769 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.093282 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.093222 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.392986 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.392957 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.702023 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.701940 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t"] Oct 30 19:53:40.702292 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.702277 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4d7edd38-7db9-4252-9996-7538308e3379" containerName="limitador" Oct 30 19:53:40.702361 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.702293 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7edd38-7db9-4252-9996-7538308e3379" containerName="limitador" Oct 30 19:53:40.702361 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.702348 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7edd38-7db9-4252-9996-7538308e3379" containerName="limitador" Oct 30 19:53:40.704642 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.704617 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.744351 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.744326 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/706e28a3-112e-4d31-bafa-57f173d640f7-data\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.744484 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.744464 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8q2z\" (UniqueName: \"kubernetes.io/projected/706e28a3-112e-4d31-bafa-57f173d640f7-kube-api-access-z8q2z\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.758649 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.758626 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:53:40.807491 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.807466 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:53:40.807739 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.807716 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 19:53:40.807837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.807729 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jk8vl\"" Oct 30 19:53:40.808894 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.808873 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t"] Oct 30 19:53:40.845282 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.845258 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/706e28a3-112e-4d31-bafa-57f173d640f7-data\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.845382 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.845356 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8q2z\" (UniqueName: \"kubernetes.io/projected/706e28a3-112e-4d31-bafa-57f173d640f7-kube-api-access-z8q2z\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.845695 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.845678 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/706e28a3-112e-4d31-bafa-57f173d640f7-data\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:40.851015 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.850985 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.906812 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:40.906791 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8q2z\" (UniqueName: \"kubernetes.io/projected/706e28a3-112e-4d31-bafa-57f173d640f7-kube-api-access-z8q2z\") pod \"postgres-7cdbf6dc44-kk59t\" (UID: \"706e28a3-112e-4d31-bafa-57f173d640f7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:41.018932 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:41.018873 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:41.385114 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:41.384947 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t"] Oct 30 19:53:41.387642 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:53:41.387617 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706e28a3_112e_4d31_bafa_57f173d640f7.slice/crio-f5b85d577c7cf609429e03674b04f7424864e393652a2243d6167f321dd55a7e WatchSource:0}: Error finding container f5b85d577c7cf609429e03674b04f7424864e393652a2243d6167f321dd55a7e: Status 404 returned error can't find the container with id f5b85d577c7cf609429e03674b04f7424864e393652a2243d6167f321dd55a7e Oct 30 19:53:41.829764 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:41.829730 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" event={"ID":"706e28a3-112e-4d31-bafa-57f173d640f7","Type":"ContainerStarted","Data":"f5b85d577c7cf609429e03674b04f7424864e393652a2243d6167f321dd55a7e"} Oct 30 19:53:42.784645 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:53:42.784610 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:53:47.939613 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:47.939586 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 19:53:48.861853 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:48.861820 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" event={"ID":"706e28a3-112e-4d31-bafa-57f173d640f7","Type":"ContainerStarted","Data":"b2b709cf8b91d597a3c2084c5a9a9d1b68fa77c5b1d5af87385e2448523e2432"} Oct 30 19:53:48.862034 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:48.861892 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:48.979754 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:48.979679 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" podStartSLOduration=2.480986212 podStartE2EDuration="8.97966556s" totalImagesPullingTime="6.498679348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:48.978723559 +0000 UTC m=+723.822395181" watchObservedRunningTime="2026-10-30 19:53:48.97966556 +0000 UTC m=+723.823337188" Oct 30 19:53:54.894253 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:54.894224 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kk59t" Oct 30 19:53:55.528121 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:55.528091 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.750567 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:53:55.750533 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.900643 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.900611 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:06.903653 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.903634 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:06.937913 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.937887 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.956248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.956210 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:54:06.956248 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.956210 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:54:06.956416 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:06.956278 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:54:07.005925 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.005900 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wdmrl\"" Oct 30 19:54:07.007318 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.007298 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:07.053629 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.053601 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq98d\" (UniqueName: \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.053781 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.053652 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.053857 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.053824 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.154443 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.154361 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.154443 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.154419 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.154652 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.154484 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq98d\" (UniqueName: \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.157035 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.156997 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.157168 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.156997 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.187364 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.187334 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:07.215971 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.215941 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq98d\" (UniqueName: \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\") pod \"maas-controller-577f596567-ttsgq\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.216682 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.216656 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:07.456149 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.456063 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 19:54:07.459192 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.459176 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.470787 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.470760 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:07.520327 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.520294 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 19:54:07.556368 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.556340 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.556487 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.556378 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.556487 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.556399 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxgn\" (UniqueName: \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.641830 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.641805 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:07.644273 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:54:07.644251 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed3e381_5ed4_413d_a5c5_a65d76e62e1a.slice/crio-1fae6c3808be6517578c6fe2cc97866ecbf7c5e27e9ee91226f0ecfe4bef5ea8 WatchSource:0}: Error finding container 1fae6c3808be6517578c6fe2cc97866ecbf7c5e27e9ee91226f0ecfe4bef5ea8: Status 404 returned error can't find the container with id 1fae6c3808be6517578c6fe2cc97866ecbf7c5e27e9ee91226f0ecfe4bef5ea8 Oct 30 19:54:07.657543 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.657521 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.657636 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.657565 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.657636 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.657618 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxgn\" (UniqueName: \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.660084 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.660059 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.660166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.660059 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.722404 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.722346 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxgn\" (UniqueName: \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\") pod \"maas-controller-84c66fdd7b-qz5rt\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.772601 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.772567 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:07.935444 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:07.935408 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ttsgq" event={"ID":"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a","Type":"ContainerStarted","Data":"1fae6c3808be6517578c6fe2cc97866ecbf7c5e27e9ee91226f0ecfe4bef5ea8"} Oct 30 19:54:08.146715 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:08.146144 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 19:54:08.942285 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:08.942209 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" event={"ID":"57065cb7-fc39-4f4b-8799-43c432f4dc83","Type":"ContainerStarted","Data":"f6a9e8f70407d62ba0c3ab95135347afbd7eed4e3af3af50d08c06b120dcba49"} Oct 30 19:54:11.632667 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.632589 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:11.958590 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.958548 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" event={"ID":"57065cb7-fc39-4f4b-8799-43c432f4dc83","Type":"ContainerStarted","Data":"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee"} Oct 30 19:54:11.958795 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.958675 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:11.960214 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.960181 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ttsgq" event={"ID":"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a","Type":"ContainerStarted","Data":"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946"} Oct 30 19:54:11.960336 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.960231 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:11.971983 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:11.971923 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.089096 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:12.089049 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" podStartSLOduration=1.9264728290000002 podStartE2EDuration="5.089035516s" totalImagesPullingTime="3.162562687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:12.08803571 +0000 UTC m=+746.931707338" watchObservedRunningTime="2026-10-30 19:54:12.089035516 +0000 UTC m=+746.932707144" Oct 30 19:54:12.239240 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:12.239152 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-ttsgq" podStartSLOduration=2.805382678 podStartE2EDuration="6.239138306s" totalImagesPullingTime="3.433755628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:12.238904633 +0000 UTC m=+747.082576263" watchObservedRunningTime="2026-10-30 19:54:12.239138306 +0000 UTC m=+747.082809967" Oct 30 19:54:12.784127 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:54:12.784090 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:54:16.563506 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:16.563470 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.782403 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:16.782354 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.966499 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:22.966470 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:22.966911 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:22.966545 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 19:54:23.238043 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.237958 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:23.238246 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.238197 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-ttsgq" podUID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" containerName="manager" containerID="cri-o://ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946" gracePeriod=30 Oct 30 19:54:23.474182 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.474158 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:23.492317 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.492273 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\") pod \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " Oct 30 19:54:23.492317 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.492308 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\") pod \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " Oct 30 19:54:23.492427 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.492367 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\" (UniqueName: \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\") pod \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\" (UID: \"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a\") " Oct 30 19:54:23.494376 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.494344 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs" pod "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" (UID: "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:23.494456 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.494369 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d" pod "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" (UID: "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a"). InnerVolumeSpecName "kube-api-access-hq98d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:23.494456 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.494447 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert" pod "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" (UID: "4ed3e381-5ed4-413d-a5c5-a65d76e62e1a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:23.593076 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.593038 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:23.593076 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.593077 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq98d\" (UniqueName: \"kubernetes.io/projected/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-kube-api-access-hq98d\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:23.593837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:23.593093 4524 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a-webhook-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:24.010612 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.010582 4524 generic.go:356] "Generic (PLEG): container finished" podID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" containerID="ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946" exitCode=0 Oct 30 19:54:24.011134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.010642 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-ttsgq" Oct 30 19:54:24.011134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.010645 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ttsgq" event={"ID":"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a","Type":"ContainerDied","Data":"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946"} Oct 30 19:54:24.011134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.010763 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-ttsgq" event={"ID":"4ed3e381-5ed4-413d-a5c5-a65d76e62e1a","Type":"ContainerDied","Data":"1fae6c3808be6517578c6fe2cc97866ecbf7c5e27e9ee91226f0ecfe4bef5ea8"} Oct 30 19:54:24.011134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.010786 4524 scope.go:122] "RemoveContainer" containerID="ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946" Oct 30 19:54:24.019148 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.019131 4524 scope.go:122] "RemoveContainer" containerID="ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946" Oct 30 19:54:24.019420 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:54:24.019402 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946\": container with ID starting with ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946 not found: ID does not exist" containerID="ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946" Oct 30 19:54:24.019466 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.019429 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946"} err="failed to get container status \"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946\": rpc error: code = NotFound desc = could not find container \"ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946\": container with ID starting with ad16b6d22498019b72d1a2fca3a10c4304a557d7cceac0a99dd5649da3e0d946 not found: ID does not exist" Oct 30 19:54:24.176244 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.176210 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:24.180773 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:24.180751 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-ttsgq"] Oct 30 19:54:25.297508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:25.297475 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" path="/var/lib/kubelet/pods/4ed3e381-5ed4-413d-a5c5-a65d76e62e1a/volumes" Oct 30 19:54:38.415181 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:38.415100 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.785788 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:54:42.785750 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:54:43.970512 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:43.970475 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:44.257983 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:44.257901 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.992989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:45.992957 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.272620 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.272534 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.510078 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.510047 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.988422 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.988385 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 19:54:46.988726 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.988712 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" containerName="manager" Oct 30 19:54:46.988779 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.988728 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" containerName="manager" Oct 30 19:54:46.988814 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.988794 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed3e381-5ed4-413d-a5c5-a65d76e62e1a" containerName="manager" Oct 30 19:54:46.995067 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:46.995049 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.071346 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.071309 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwh9\" (UniqueName: \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.071484 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.071373 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.071484 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.071440 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.071573 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.071556 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.103692 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.103663 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:54:47.103868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.103780 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xtx52\"" Oct 30 19:54:47.103868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.103818 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:54:47.104944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.104926 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 19:54:47.172868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.172846 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.173022 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.172877 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwh9\" (UniqueName: \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.173022 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.172907 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.173022 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.172928 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.173505 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.173476 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.175352 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.175330 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.175442 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.175354 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.236707 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.236683 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwh9\" (UniqueName: \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\") pod \"maas-api-5c48877c75-vwdcm\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.307292 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.307239 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:47.683610 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:47.683582 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 19:54:47.686577 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:54:47.686550 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba9b876_d448_409f_b43e_c4a304e4e4fd.slice/crio-be5a4a49e48c3ffabeb868a08a43e9e43ec38dc4c721001a87b5b0a071948c46 WatchSource:0}: Error finding container be5a4a49e48c3ffabeb868a08a43e9e43ec38dc4c721001a87b5b0a071948c46: Status 404 returned error can't find the container with id be5a4a49e48c3ffabeb868a08a43e9e43ec38dc4c721001a87b5b0a071948c46 Oct 30 19:54:48.111496 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:48.111424 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" event={"ID":"aba9b876-d448-409f-b43e-c4a304e4e4fd","Type":"ContainerStarted","Data":"be5a4a49e48c3ffabeb868a08a43e9e43ec38dc4c721001a87b5b0a071948c46"} Oct 30 19:54:50.122551 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:50.122466 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" event={"ID":"aba9b876-d448-409f-b43e-c4a304e4e4fd","Type":"ContainerStarted","Data":"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8"} Oct 30 19:54:50.122989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:50.122595 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:54:50.250932 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:50.250878 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" podStartSLOduration=2.4360793210000002 podStartE2EDuration="4.250863536s" totalImagesPullingTime="1.814784215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:50.250210159 +0000 UTC m=+785.093881789" watchObservedRunningTime="2026-10-30 19:54:50.250863536 +0000 UTC m=+785.094535166" Oct 30 19:54:56.131874 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:54:56.131843 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 19:55:12.785219 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:55:12.785177 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:55:31.725038 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:31.724995 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.124301 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:32.124224 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.765795 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:38.765754 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.784842 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:55:42.784809 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:55:48.610155 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:48.610116 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.209493 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.209457 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4"] Oct 30 19:55:52.213045 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.213022 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.269010 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.268984 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lfm4p\"" Oct 30 19:55:52.284326 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284289 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.284444 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284423 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.284565 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284545 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.284641 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284615 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.284716 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284641 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8f6\" (UniqueName: \"kubernetes.io/projected/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-kube-api-access-kd8f6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.284716 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.284673 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.318588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.318567 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:55:52.318691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.318588 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:55:52.318691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.318602 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:55:52.320151 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.320129 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4"] Oct 30 19:55:52.385478 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385446 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385575 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385493 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385575 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385560 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385691 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385676 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385759 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385749 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385774 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8f6\" (UniqueName: \"kubernetes.io/projected/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-kube-api-access-kd8f6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385897 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.385999 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.385981 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.386125 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.386105 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.387842 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.387827 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.388007 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.387991 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.433583 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.433559 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.452809 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.452776 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8f6\" (UniqueName: \"kubernetes.io/projected/6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9-kube-api-access-kd8f6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mnfw4\" (UID: \"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.552219 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.552164 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:52.947019 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:52.946985 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4"] Oct 30 19:55:52.950335 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:55:52.950307 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6493447a_ee7e_42b6_b1fa_9e5b0fc7d6c9.slice/crio-4803297805ea82cc6bc24baf1db956d3500bdea253624747f2f66a3554f9d5b6 WatchSource:0}: Error finding container 4803297805ea82cc6bc24baf1db956d3500bdea253624747f2f66a3554f9d5b6: Status 404 returned error can't find the container with id 4803297805ea82cc6bc24baf1db956d3500bdea253624747f2f66a3554f9d5b6 Oct 30 19:55:53.364948 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:53.364878 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" event={"ID":"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9","Type":"ContainerStarted","Data":"4803297805ea82cc6bc24baf1db956d3500bdea253624747f2f66a3554f9d5b6"} Oct 30 19:55:55.374835 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.374794 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" event={"ID":"6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9","Type":"ContainerStarted","Data":"1771b11fbbcca112543f08b14898c695a686baa4f056f49885a11944e7fb6424"} Oct 30 19:55:55.508715 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.508655 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" podStartSLOduration=2.014933626 podStartE2EDuration="3.508641515s" totalImagesPullingTime="1.493707889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:55.507304171 +0000 UTC m=+850.350975862" watchObservedRunningTime="2026-10-30 19:55:55.508641515 +0000 UTC m=+850.352313169" Oct 30 19:55:55.552900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.552874 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:55:55.864975 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.864936 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5"] Oct 30 19:55:55.868320 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.868299 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914221 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914181 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914308 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhn5\" (UniqueName: \"kubernetes.io/projected/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-kube-api-access-fhhn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914425 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914399 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914470 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914444 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914520 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914471 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.914584 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.914564 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:55.922018 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.921964 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:55:55.973399 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:55.973369 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5"] Oct 30 19:55:56.015871 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.015846 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhn5\" (UniqueName: \"kubernetes.io/projected/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-kube-api-access-fhhn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.015877 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.015905 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016014 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016180 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016053 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016180 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016106 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016384 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016365 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016406 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.016445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.016417 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.018108 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.018087 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.018565 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.018545 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.086967 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.086940 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhn5\" (UniqueName: \"kubernetes.io/projected/e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1-kube-api-access-fhhn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7d7d5\" (UID: \"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.163429 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.163401 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.180288 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.180266 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:56.487197 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.487099 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.571255 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:56.571230 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5"] Oct 30 19:55:56.574531 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:55:56.574503 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7487f7e_ed57_4a3a_aa9d_dc78d5f13da1.slice/crio-8ab74db7fd07d63c6e9733b2661b503ebfdb3e0ec948a5d43f4fbc08ab4c644d WatchSource:0}: Error finding container 8ab74db7fd07d63c6e9733b2661b503ebfdb3e0ec948a5d43f4fbc08ab4c644d: Status 404 returned error can't find the container with id 8ab74db7fd07d63c6e9733b2661b503ebfdb3e0ec948a5d43f4fbc08ab4c644d Oct 30 19:55:57.384417 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:57.384385 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" event={"ID":"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1","Type":"ContainerStarted","Data":"66ac9884131a14041275183d4cafba9f320b4eaa36e019d01f5ac2a1bbe9fbcf"} Oct 30 19:55:57.384417 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:57.384431 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" event={"ID":"e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1","Type":"ContainerStarted","Data":"8ab74db7fd07d63c6e9733b2661b503ebfdb3e0ec948a5d43f4fbc08ab4c644d"} Oct 30 19:55:58.180574 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:58.180543 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:55:59.647236 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:55:59.647201 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:02.552638 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:02.552603 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:56:02.564898 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:02.564871 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:56:02.705708 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:02.705659 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" podStartSLOduration=7.129772058 podStartE2EDuration="7.705645231s" totalImagesPullingTime="575.873173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:57.507224797 +0000 UTC m=+852.350896426" watchObservedRunningTime="2026-10-30 19:56:02.705645231 +0000 UTC m=+857.549316859" Oct 30 19:56:03.420111 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.420085 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" Oct 30 19:56:03.524906 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.524875 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc"] Oct 30 19:56:03.528346 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.528325 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.573613 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573577 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7549f\" (UniqueName: \"kubernetes.io/projected/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-kube-api-access-7549f\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573683 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573758 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573785 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573891 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.573925 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.578889 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.578860 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc"] Oct 30 19:56:03.589091 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.589067 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:56:03.674712 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674621 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.674712 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674658 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.674712 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674681 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7549f\" (UniqueName: \"kubernetes.io/projected/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-kube-api-access-7549f\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.674982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674726 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.674982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674755 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.674982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.674771 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.675143 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.675086 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.675201 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.675138 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.675242 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.675198 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.676629 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.676605 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.677445 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.677420 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.677570 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.677422 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.736634 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.736604 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7549f\" (UniqueName: \"kubernetes.io/projected/6f2e017c-f9f6-4850-a8c1-63e34c1706f9-kube-api-access-7549f\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc\" (UID: \"6f2e017c-f9f6-4850-a8c1-63e34c1706f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:03.841108 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:03.841074 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:04.221647 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:04.221613 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc"] Oct 30 19:56:04.223854 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:56:04.223833 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2e017c_f9f6_4850_a8c1_63e34c1706f9.slice/crio-4837ec259d2022d2699709258632a2988689b42f72b27aec039798630b134a8d WatchSource:0}: Error finding container 4837ec259d2022d2699709258632a2988689b42f72b27aec039798630b134a8d: Status 404 returned error can't find the container with id 4837ec259d2022d2699709258632a2988689b42f72b27aec039798630b134a8d Oct 30 19:56:04.414116 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:04.414083 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" event={"ID":"6f2e017c-f9f6-4850-a8c1-63e34c1706f9","Type":"ContainerStarted","Data":"4837ec259d2022d2699709258632a2988689b42f72b27aec039798630b134a8d"} Oct 30 19:56:05.421268 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:05.421234 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" event={"ID":"6f2e017c-f9f6-4850-a8c1-63e34c1706f9","Type":"ContainerStarted","Data":"ef2bf4f7d61c677e669c8a60ff35dc02fc70edf76c3cccc2e8966448469b56ae"} Oct 30 19:56:05.540534 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:05.540478 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" podStartSLOduration=1.9418065690000001 podStartE2EDuration="2.540463279s" totalImagesPullingTime="598.65671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:05.539677601 +0000 UTC m=+860.383349242" watchObservedRunningTime="2026-10-30 19:56:05.540463279 +0000 UTC m=+860.384134912" Oct 30 19:56:05.841624 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:05.841550 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:06.180600 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:06.180575 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:56:06.192630 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:06.192609 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:56:06.437478 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:06.437406 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7d7d5" Oct 30 19:56:07.561996 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:07.561966 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.537671 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:12.537634 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.786193 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:56:12.786149 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:56:13.841897 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:13.841866 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:13.854583 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:13.854560 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:14.471195 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:14.471168 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc" Oct 30 19:56:15.136935 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.136893 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm"] Oct 30 19:56:15.142784 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.142759 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.202333 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.202297 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:56:15.253283 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.253257 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm"] Oct 30 19:56:15.264385 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264354 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.264513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264391 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.264513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264420 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqtz\" (UniqueName: \"kubernetes.io/projected/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-kube-api-access-snqtz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.264587 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264541 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.264661 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264606 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.264661 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.264628 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365254 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365223 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365254 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365257 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365301 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365327 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365353 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqtz\" (UniqueName: \"kubernetes.io/projected/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-kube-api-access-snqtz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365418 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365682 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365621 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365790 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365678 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.365790 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.365765 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.367212 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.367190 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.367988 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.367971 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.368158 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.368144 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.426241 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.426210 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqtz\" (UniqueName: \"kubernetes.io/projected/36dc7fbd-ec30-4beb-8b63-6c82b7948da4-kube-api-access-snqtz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-x4wvm\" (UID: \"36dc7fbd-ec30-4beb-8b63-6c82b7948da4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.456440 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.456416 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:15.838659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:15.838584 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm"] Oct 30 19:56:15.842162 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:56:15.842137 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc7fbd_ec30_4beb_8b63_6c82b7948da4.slice/crio-d351a64e200cb7601bf7d36b8c348cb35b767edb9be5fc15c9fb35e48fa9088c WatchSource:0}: Error finding container d351a64e200cb7601bf7d36b8c348cb35b767edb9be5fc15c9fb35e48fa9088c: Status 404 returned error can't find the container with id d351a64e200cb7601bf7d36b8c348cb35b767edb9be5fc15c9fb35e48fa9088c Oct 30 19:56:16.470085 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:16.470059 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" event={"ID":"36dc7fbd-ec30-4beb-8b63-6c82b7948da4","Type":"ContainerStarted","Data":"d351a64e200cb7601bf7d36b8c348cb35b767edb9be5fc15c9fb35e48fa9088c"} Oct 30 19:56:17.477027 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:17.476994 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" event={"ID":"36dc7fbd-ec30-4beb-8b63-6c82b7948da4","Type":"ContainerStarted","Data":"8b57a69574b474b020651b2adeb2338178e1c2159f61b8fe7919518695fc5c14"} Oct 30 19:56:17.597585 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:17.597531 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" podStartSLOduration=2.011810248 podStartE2EDuration="2.597519623s" totalImagesPullingTime="585.709375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:17.595990126 +0000 UTC m=+872.439661756" watchObservedRunningTime="2026-10-30 19:56:17.597519623 +0000 UTC m=+872.441191251" Oct 30 19:56:18.457089 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:18.457054 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:25.456902 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:25.456870 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:25.469447 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:25.469427 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:25.522342 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:25.522322 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" Oct 30 19:56:31.970883 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:31.970849 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 19:56:31.974290 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:31.974274 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.020974 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.020952 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 19:56:32.024111 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.024092 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.101894 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.101860 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.101893 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.101918 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp4k\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.101997 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102054 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.102038 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102268 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.102120 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102268 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.102255 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102332 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.102277 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.102332 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.102305 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.111215 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.111197 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\"" Oct 30 19:56:32.152196 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.152174 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 19:56:32.187767 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.187729 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 19:56:32.203267 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203241 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203420 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203275 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203420 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203293 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203420 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203311 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp4k\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203598 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203441 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203598 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203486 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203598 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203515 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203642 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203727 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203757 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.203957 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203823 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52zf\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203957 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203868 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203957 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203900 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.203957 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203945 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203968 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.204166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.203994 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.204166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204027 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204084 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204071 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.204407 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204197 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204407 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204279 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204523 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204408 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.204585 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.204515 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.206972 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.206952 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.207077 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.206969 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.266186 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.266126 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.266186 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.266155 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp4k\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2ljcs\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.286971 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.286950 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:32.304865 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.304838 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.304977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.304875 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.304977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.304898 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.304977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.304914 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.304977 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.304944 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305012 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305038 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305062 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305183 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305130 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52zf\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305392 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305271 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305549 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305515 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305657 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305563 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305795 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305772 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.305865 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.305823 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.307159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.307140 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.307412 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.307396 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.367572 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.367544 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52zf\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.367734 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.367626 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-745qj\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.636440 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.636369 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:32.716076 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:32.716049 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 19:56:32.717971 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:56:32.717947 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf7171d_7aef_46b7_9af6_d3e0773a41b2.slice/crio-cc7511eb33e00ccee698927772d146727a7a22bcee5151fea0d6151fd3bea605 WatchSource:0}: Error finding container cc7511eb33e00ccee698927772d146727a7a22bcee5151fea0d6151fd3bea605: Status 404 returned error can't find the container with id cc7511eb33e00ccee698927772d146727a7a22bcee5151fea0d6151fd3bea605 Oct 30 19:56:33.010572 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:33.010548 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 19:56:33.013079 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:56:33.013050 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3517d6d9_a773_4377_bff2_8ef3a0b24893.slice/crio-7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592 WatchSource:0}: Error finding container 7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592: Status 404 returned error can't find the container with id 7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592 Oct 30 19:56:33.548175 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:33.548137 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" event={"ID":"bbf7171d-7aef-46b7-9af6-d3e0773a41b2","Type":"ContainerStarted","Data":"cc7511eb33e00ccee698927772d146727a7a22bcee5151fea0d6151fd3bea605"} Oct 30 19:56:33.549231 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:33.549203 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" event={"ID":"3517d6d9-a773-4377-bff2-8ef3a0b24893","Type":"ContainerStarted","Data":"7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592"} Oct 30 19:56:36.248528 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.248460 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.469861 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.469824 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.991719 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.991669 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:36.991820 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.991754 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:36.991820 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.991784 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:36.999127 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.999091 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:36.999227 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.999208 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:36.999291 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:36.999275 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:37.592721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:37.588618 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" event={"ID":"bbf7171d-7aef-46b7-9af6-d3e0773a41b2","Type":"ContainerStarted","Data":"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072"} Oct 30 19:56:37.603725 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:37.603000 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" event={"ID":"3517d6d9-a773-4377-bff2-8ef3a0b24893","Type":"ContainerStarted","Data":"818c95e9c1a6b040872248460866df4ed8c89dca21badea782371898f576d73b"} Oct 30 19:56:37.636854 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:37.636817 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:37.715744 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:37.715670 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podStartSLOduration=2.437334524 podStartE2EDuration="6.715655954s" totalImagesPullingTime="4.27832143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:37.715393589 +0000 UTC m=+892.559065219" watchObservedRunningTime="2026-10-30 19:56:37.715655954 +0000 UTC m=+892.559327583" Oct 30 19:56:38.292724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:38.291997 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:38.294724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:38.293540 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:38.294724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:38.293592 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:38.638776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:38.638685 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:38.638776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:38.638756 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:39.173330 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:39.173244 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:39.291723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:39.288070 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:39.291723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:39.288131 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:39.637293 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:39.637207 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:39.637293 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:39.637260 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:40.290720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:40.290087 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:40.290720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:40.290150 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:40.637612 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:40.637522 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:40.637612 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:40.637588 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:41.288722 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:41.288086 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:41.288722 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:41.288148 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:41.637359 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:41.637267 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:41.637359 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:41.637335 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:42.291724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:42.288080 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:42.291724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:42.288146 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:42.637715 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:42.637621 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:42.638094 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:42.637683 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:42.814306 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:56:42.814270 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:56:43.288256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:43.288210 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:43.288432 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:43.288281 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:43.637539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:43.637464 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:43.637539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:43.637527 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:44.287953 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:44.287913 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:56:44.288340 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:44.287983 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:56:44.636842 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:44.636747 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-745qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:56:44.636842 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:44.636811 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:56:45.507848 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.507816 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:45.509327 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.509308 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:45.510282 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.510262 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 19:56:45.641070 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.641035 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:45.641325 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.641306 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:45.641929 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.641913 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 19:56:45.683398 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:45.683340 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podStartSLOduration=10.707472344 podStartE2EDuration="14.683322116s" totalImagesPullingTime="3.975849772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:37.840120739 +0000 UTC m=+892.683792368" watchObservedRunningTime="2026-10-30 19:56:45.683322116 +0000 UTC m=+900.526993746" Oct 30 19:56:46.213471 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:46.213437 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.517181 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:56:48.517125 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.681860 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:02.681822 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.910551 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:02.910520 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.940023 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:10.939990 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.458745 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:11.458688 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.685241 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:11.685204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.785201 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:12.785166 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:57:13.170650 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:13.170620 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.438245 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:13.438169 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.774297 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:14.774246 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.047082 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:15.047008 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.524890 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:39.524818 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.953362 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:40.953295 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.221081 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:41.221000 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.654066 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:41.654033 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.891409 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:41.891353 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.129134 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.129047 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 19:57:42.131029 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.131003 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.132533 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.132517 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.199260 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.198379 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 19:57:42.244562 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244529 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244562 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244563 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244583 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244661 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244794 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244715 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244908 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244832 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244944 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244920 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7vb\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244983 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244949 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.244983 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.244965 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.248412 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.248388 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3ccc9f-openshift-default-dockercfg-sn7l4\"" Oct 30 19:57:42.345479 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345451 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345479 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345481 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345503 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345528 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345546 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345581 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345623 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7vb\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345645 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.345778 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345661 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.346155 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.345975 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.346214 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.346162 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.346279 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.346220 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.346343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.346277 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.346458 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.346440 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.347953 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.347919 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.348166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.348149 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.423513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.423480 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7vb\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.430954 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.430930 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.445658 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.445636 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:42.476579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.476553 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.760221 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.760122 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.785362 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:42.785320 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:57:42.844915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.844882 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 19:57:42.849275 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.849253 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.884941 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.884907 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 19:57:42.898495 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:57:42.898463 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a61d0c_c691_4c79_81a5_ac5940b0c4a0.slice/crio-a9213af92afdedcf954fdca956c9804bfae92f765da0f6359b587913a7a1ef33 WatchSource:0}: Error finding container a9213af92afdedcf954fdca956c9804bfae92f765da0f6359b587913a7a1ef33: Status 404 returned error can't find the container with id a9213af92afdedcf954fdca956c9804bfae92f765da0f6359b587913a7a1ef33 Oct 30 19:57:42.901207 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.901173 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:42.901313 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.901243 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:42.901313 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.901274 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:42.904279 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.904189 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-779921-openshift-default-dockercfg-rs78c\"" Oct 30 19:57:42.912527 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.912501 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 19:57:42.957655 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957632 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.957780 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957664 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zzj\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.957831 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957775 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.957906 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957885 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.957970 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957929 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.958023 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.957967 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.958095 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.958081 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.958140 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.958122 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:42.958217 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:42.958152 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.009532 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.009500 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.059125 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059047 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059125 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059081 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059125 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059109 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059127 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zzj\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059187 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059216 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059247 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059275 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059387 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059321 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059730 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059603 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059730 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059661 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.059837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.059786 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.060280 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.060255 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.060539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.060518 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.061547 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.061521 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.062205 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.062188 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.120566 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.120538 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.120720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.120656 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zzj\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.162797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.162774 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:43.244242 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.244217 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.531454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.531427 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 19:57:43.542451 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.542412 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.550960 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.550921 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.598751 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.598729 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebb42-openshift-default-dockercfg-frbd5\"" Oct 30 19:57:43.614298 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.614271 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 19:57:43.615797 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.615777 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 19:57:43.620842 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:57:43.620814 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280d856f_7580_44f5_a96e_f3fbc4355780.slice/crio-e3951e30b6ecb5efbefd0808b3056fd28dd4a61989deb74b4c81890151016bdd WatchSource:0}: Error finding container e3951e30b6ecb5efbefd0808b3056fd28dd4a61989deb74b4c81890151016bdd: Status 404 returned error can't find the container with id e3951e30b6ecb5efbefd0808b3056fd28dd4a61989deb74b4c81890151016bdd Oct 30 19:57:43.623473 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.623444 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:43.623559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.623504 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:43.623559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.623533 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:43.665175 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665147 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665268 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665197 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665351 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665331 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665390 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665374 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665442 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665404 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665517 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclhs\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665654 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665570 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665654 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665589 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.665808 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.665660 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.766914 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.766871 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767100 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.766924 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767100 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.766952 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767100 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.766991 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclhs\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767273 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767122 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767273 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767169 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767486 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767624 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767534 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767624 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767556 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767624 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767587 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.767880 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.767627 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.768582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.768042 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.768582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.768085 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.768582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.768151 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.770575 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.770551 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.770662 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.770580 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.802017 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.801960 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.829561 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.829527 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclhs\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.832581 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.832551 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.864582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.864559 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:43.910135 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.910090 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerStarted","Data":"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3"} Oct 30 19:57:43.910301 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.910153 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerStarted","Data":"e3951e30b6ecb5efbefd0808b3056fd28dd4a61989deb74b4c81890151016bdd"} Oct 30 19:57:43.912827 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.912274 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerStarted","Data":"e2cea8dcf26cb1fa948d7853efd3e96147011ca9f3cb52b303f37ffcc13234d6"} Oct 30 19:57:43.912827 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:43.912327 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerStarted","Data":"a9213af92afdedcf954fdca956c9804bfae92f765da0f6359b587913a7a1ef33"} Oct 30 19:57:44.034194 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.034158 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.034391 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.034312 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podStartSLOduration=2.034297856 podStartE2EDuration="2.034297856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:44.032012844 +0000 UTC m=+958.875684474" watchObservedRunningTime="2026-10-30 19:57:44.034297856 +0000 UTC m=+958.877969479" Oct 30 19:57:44.158524 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.158458 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podStartSLOduration=2.158439862 podStartE2EDuration="2.158439862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:44.156534387 +0000 UTC m=+959.000206017" watchObservedRunningTime="2026-10-30 19:57:44.158439862 +0000 UTC m=+959.002111494" Oct 30 19:57:44.163614 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.163580 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:44.168507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.168483 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:44.288579 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.288528 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 19:57:44.292551 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:57:44.292512 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4bf278_0627_4ef8_b1e0_f101c91f0549.slice/crio-0ef6910ede3b0ef885a2a0c9178704c5c9b7e234a86d9e95de78f3929e48fa50 WatchSource:0}: Error finding container 0ef6910ede3b0ef885a2a0c9178704c5c9b7e234a86d9e95de78f3929e48fa50: Status 404 returned error can't find the container with id 0ef6910ede3b0ef885a2a0c9178704c5c9b7e234a86d9e95de78f3929e48fa50 Oct 30 19:57:44.296349 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.296319 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:44.296424 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.296375 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:44.296424 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.296405 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:44.445949 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.445874 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:44.450771 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.450744 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:44.918730 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.918671 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerStarted","Data":"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc"} Oct 30 19:57:44.919179 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.919155 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:44.919312 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.919213 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:44.919312 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.919239 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerStarted","Data":"0ef6910ede3b0ef885a2a0c9178704c5c9b7e234a86d9e95de78f3929e48fa50"} Oct 30 19:57:44.919842 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.919818 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:57:44.919945 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:44.919907 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:57:45.041723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:45.041655 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podStartSLOduration=2.041642596 podStartE2EDuration="2.041642596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:45.039799584 +0000 UTC m=+959.883471238" watchObservedRunningTime="2026-10-30 19:57:45.041642596 +0000 UTC m=+959.885314224" Oct 30 19:57:45.865419 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:45.865381 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:45.870409 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:45.870379 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:45.923850 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:45.923825 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:45.924477 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:45.924459 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:57:46.737327 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:46.737291 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.995664 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:46.995587 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.442116 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:47.442088 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.755947 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:47.755868 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.992327 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:47.992291 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.249667 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:48.249640 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.505283 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:48.505205 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.763117 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:48.763039 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.993767 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:48.993736 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.282924 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:49.282896 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.519589 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:49.519558 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.760058 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:49.760027 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.008004 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:50.007967 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.308653 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:51.308618 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.535438 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:51.535405 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.761158 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:51.761125 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.246689 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:52.244314 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.531428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:52.531342 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.121538 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.121503 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 19:57:53.126045 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.126024 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.228821 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.228789 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-407632-serving-cert\"" Oct 30 19:57:53.230659 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.230634 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 19:57:53.259651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.259623 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.259651 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.259665 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2tw\" (UniqueName: \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.259939 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.259719 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.259939 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.259900 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.361395 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.361358 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.361567 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.361405 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2tw\" (UniqueName: \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.361567 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.361437 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.361567 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:53.361476 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-serving-cert: secret "maas-api-e2e-worker-w5-407632-serving-cert" not found Oct 30 19:57:53.361567 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:53.361542 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls podName:d842657a-9d13-4742-b136-adc407e85395 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:53.861526871 +0000 UTC m=+968.705198491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls") pod "maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" (UID: "d842657a-9d13-4742-b136-adc407e85395") : secret "maas-api-e2e-worker-w5-407632-serving-cert" not found Oct 30 19:57:53.361836 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.361594 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.362146 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.362113 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.363869 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.363850 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.423225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.423186 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2tw\" (UniqueName: \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.525098 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.525063 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.777996 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.777863 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.866144 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.866109 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:53.868516 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:53.868488 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") pod \"maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:54.038924 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:54.038856 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:54.415142 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:54.415115 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 19:57:54.418174 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:57:54.418147 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd842657a_9d13_4742_b136_adc407e85395.slice/crio-2f710a0913670dd52b392ec3c18221e618b6007b10abbbb7babfe9670e5b9d83 WatchSource:0}: Error finding container 2f710a0913670dd52b392ec3c18221e618b6007b10abbbb7babfe9670e5b9d83: Status 404 returned error can't find the container with id 2f710a0913670dd52b392ec3c18221e618b6007b10abbbb7babfe9670e5b9d83 Oct 30 19:57:54.964042 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:54.964009 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" event={"ID":"d842657a-9d13-4742-b136-adc407e85395","Type":"ContainerStarted","Data":"2f710a0913670dd52b392ec3c18221e618b6007b10abbbb7babfe9670e5b9d83"} Oct 30 19:57:55.969354 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:55.969322 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" event={"ID":"d842657a-9d13-4742-b136-adc407e85395","Type":"ContainerStarted","Data":"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d"} Oct 30 19:57:55.969754 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:55.969395 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:57:56.088941 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:56.088882 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" podStartSLOduration=2.300519211 podStartE2EDuration="3.088864794s" totalImagesPullingTime="788.345583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:56.08727738 +0000 UTC m=+970.930949010" watchObservedRunningTime="2026-10-30 19:57:56.088864794 +0000 UTC m=+970.932536424" Oct 30 19:57:57.640241 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:57.640208 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.868153 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:57.868099 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.402084 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.402049 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 19:57:58.405589 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.405567 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.462577 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.462558 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-779921-serving-cert\"" Oct 30 19:57:58.508588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.508562 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.508752 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.508598 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwtb\" (UniqueName: \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.508752 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.508639 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.508833 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.508770 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.514163 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.514139 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 19:57:58.609837 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.609809 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.609982 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.609844 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwtb\" (UniqueName: \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.609982 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:58.609958 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-serving-cert: secret "maas-api-e2e-worker-w0-779921-serving-cert" not found Oct 30 19:57:58.610089 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.609982 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.610089 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:57:58.610015 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls podName:fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:59.109999048 +0000 UTC m=+973.953670655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls") pod "maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" (UID: "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212") : secret "maas-api-e2e-worker-w0-779921-serving-cert" not found Oct 30 19:57:58.610089 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.610063 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.610654 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.610633 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.612417 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.612393 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.673912 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.673862 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwtb\" (UniqueName: \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:58.740160 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.740132 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.956041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:58.955957 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.114750 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:59.114685 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:59.117115 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:59.117083 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") pod \"maas-api-e2e-worker-w0-779921-98c756c89-k9xlq\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:59.320900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:59.320818 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:57:59.695970 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:59.695937 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 19:57:59.697930 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:57:59.697900 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5f5ae7_0edb_40e5_a1aa_8190b3c8a212.slice/crio-0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3 WatchSource:0}: Error finding container 0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3: Status 404 returned error can't find the container with id 0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3 Oct 30 19:57:59.989950 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:57:59.989868 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" event={"ID":"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212","Type":"ContainerStarted","Data":"0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3"} Oct 30 19:58:00.397031 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:00.396991 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.619308 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:00.619272 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.997791 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:00.997752 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" event={"ID":"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212","Type":"ContainerStarted","Data":"875ba5e0a303050788b7fc86a6eef1f7b475d81ca64c6ada518bb9731bfd9fa2"} Oct 30 19:58:00.998292 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:00.997907 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:58:01.119594 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:01.119525 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" podStartSLOduration=2.336429766 podStartE2EDuration="3.119505454s" totalImagesPullingTime="783.075688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:01.119142999 +0000 UTC m=+975.962814628" watchObservedRunningTime="2026-10-30 19:58:01.119505454 +0000 UTC m=+975.963177083" Oct 30 19:58:01.979299 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:01.979270 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 19:58:03.245080 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.244993 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.493507 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.493475 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 19:58:03.497388 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.497327 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.510041 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.510019 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.560065 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.560037 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 19:58:03.607756 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.607728 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-3ccc9f-serving-cert\"" Oct 30 19:58:03.693854 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.693824 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6922z\" (UniqueName: \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.694021 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.693861 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.694067 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.694020 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.694067 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.694038 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.790201 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.790105 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.795822 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.795774 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.795989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.795826 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.795989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.795907 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6922z\" (UniqueName: \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.795989 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.795936 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.797454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.797422 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.799888 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.799861 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.799985 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.799865 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:03.858203 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:03.858173 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6922z\" (UniqueName: \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\") pod \"maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:04.015319 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:04.015283 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.110588 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:04.110517 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:04.496637 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:04.496607 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 19:58:04.498165 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:58:04.498136 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e02b61_5689_482a_aeb4_544bd5e88d50.slice/crio-0c33e7d01fe51835eced3958d74b78451904f455b47d4b9f7bcf8fc3118f2599 WatchSource:0}: Error finding container 0c33e7d01fe51835eced3958d74b78451904f455b47d4b9f7bcf8fc3118f2599: Status 404 returned error can't find the container with id 0c33e7d01fe51835eced3958d74b78451904f455b47d4b9f7bcf8fc3118f2599 Oct 30 19:58:05.017649 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:05.017600 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" event={"ID":"a0e02b61-5689-482a-aeb4-544bd5e88d50","Type":"ContainerStarted","Data":"0c33e7d01fe51835eced3958d74b78451904f455b47d4b9f7bcf8fc3118f2599"} Oct 30 19:58:06.024159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:06.024119 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" event={"ID":"a0e02b61-5689-482a-aeb4-544bd5e88d50","Type":"ContainerStarted","Data":"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc"} Oct 30 19:58:06.024557 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:06.024267 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:06.128503 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:06.128471 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:06.146889 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:06.146841 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" podStartSLOduration=2.302918008 podStartE2EDuration="3.146829264s" totalImagesPullingTime="843.911256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:06.146370743 +0000 UTC m=+980.990042373" watchObservedRunningTime="2026-10-30 19:58:06.146829264 +0000 UTC m=+980.990500893" Oct 30 19:58:06.387834 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:06.387756 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.021317 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:07.021285 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 19:58:08.601995 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.601956 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 19:58:08.605528 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.605504 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.641589 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.641556 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtc8\" (UniqueName: \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.641723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.641604 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.641723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.641624 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.641803 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.641763 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.660029 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.660006 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f84d85-serving-cert\"" Oct 30 19:58:08.711456 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.711429 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 19:58:08.742593 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.742563 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.742770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.742731 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtc8\" (UniqueName: \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.742843 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:08.742763 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-serving-cert: secret "maas-api-e2e-worker-w3-f84d85-serving-cert" not found Oct 30 19:58:08.742900 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:08.742843 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls podName:22242480-dae8-4d0a-9d77-69742ef38d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:58:09.242823869 +0000 UTC m=+984.086495477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls") pod "maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" (UID: "22242480-dae8-4d0a-9d77-69742ef38d8e") : secret "maas-api-e2e-worker-w3-f84d85-serving-cert" not found Oct 30 19:58:08.742900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.742766 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.742900 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.742888 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.743478 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.743450 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.745428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.745408 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.805453 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.805425 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtc8\" (UniqueName: \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:08.877190 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:08.877123 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.105850 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.105816 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.247710 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.247670 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:09.250409 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.250377 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:09.340373 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.340339 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.521559 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.521479 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:09.905296 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:09.905212 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 19:58:09.907956 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:58:09.907926 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22242480_dae8_4d0a_9d77_69742ef38d8e.slice/crio-f0aca8867a68a4fa1d1f2f3800711e03a2c684289da98e1340a9c6fa9b1e1e96 WatchSource:0}: Error finding container f0aca8867a68a4fa1d1f2f3800711e03a2c684289da98e1340a9c6fa9b1e1e96: Status 404 returned error can't find the container with id f0aca8867a68a4fa1d1f2f3800711e03a2c684289da98e1340a9c6fa9b1e1e96 Oct 30 19:58:10.065874 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:10.065838 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" event={"ID":"22242480-dae8-4d0a-9d77-69742ef38d8e","Type":"ContainerStarted","Data":"f0aca8867a68a4fa1d1f2f3800711e03a2c684289da98e1340a9c6fa9b1e1e96"} Oct 30 19:58:10.691955 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:10.691930 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.100483 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:11.100389 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" event={"ID":"22242480-dae8-4d0a-9d77-69742ef38d8e","Type":"ContainerStarted","Data":"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66"} Oct 30 19:58:11.100915 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:11.100784 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:11.225582 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:11.225532 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" podStartSLOduration=2.468711042 podStartE2EDuration="3.225519707s" totalImagesPullingTime="756.808665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:11.225166789 +0000 UTC m=+986.068838420" watchObservedRunningTime="2026-10-30 19:58:11.225519707 +0000 UTC m=+986.069191336" Oct 30 19:58:11.459468 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:11.459429 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.713533 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:11.713443 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.037122 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:12.037054 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 19:58:12.821055 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:12.821001 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:58:14.208374 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:14.208332 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:16.869069 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:16.868977 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.115149 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:17.115112 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.124441 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:17.124384 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 19:58:18.828630 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.828590 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 19:58:18.832273 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.832238 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:18.886721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.885848 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-d67ea9-serving-cert\"" Oct 30 19:58:18.901107 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.900727 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:18.901107 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.900803 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:18.901107 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.900928 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97dp\" (UniqueName: \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:18.901107 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.900963 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:18.938454 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:18.938423 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 19:58:19.005482 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.005250 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97dp\" (UniqueName: \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.005482 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.005315 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.005482 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.005395 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.005482 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.005442 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.008020 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.007295 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.008554 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:19.008259 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-serving-cert: secret "maas-api-e2e-worker-w2-d67ea9-serving-cert" not found Oct 30 19:58:19.008554 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:19.008349 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls podName:6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1 nodeName:}" failed. No retries permitted until 2026-10-30 19:58:19.50832702 +0000 UTC m=+994.351998643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls") pod "maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" (UID: "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1") : secret "maas-api-e2e-worker-w2-d67ea9-serving-cert" not found Oct 30 19:58:19.024461 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.024377 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.068670 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.068608 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97dp\" (UniqueName: \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.097333 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.097191 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.309834 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.309778 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.510258 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.510229 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.512999 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.512968 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:19.537490 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.537462 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.745574 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:19.745549 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:20.131973 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.131941 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 19:58:20.132824 ip-10-0-130-41 kubenswrapper[4524]: W1030 19:58:20.132802 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc92a9d_b2a0_4cd9_bfb2_1a702fc66fd1.slice/crio-4e3f7043e69ec25c180b9ad9e7e4668fe25f6eab1ceaf987665c5fe42ad2842f WatchSource:0}: Error finding container 4e3f7043e69ec25c180b9ad9e7e4668fe25f6eab1ceaf987665c5fe42ad2842f: Status 404 returned error can't find the container with id 4e3f7043e69ec25c180b9ad9e7e4668fe25f6eab1ceaf987665c5fe42ad2842f Oct 30 19:58:20.134679 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.134663 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:58:20.169149 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.169126 4524 generic.go:356] "Generic (PLEG): container finished" podID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerID="e2cea8dcf26cb1fa948d7853efd3e96147011ca9f3cb52b303f37ffcc13234d6" exitCode=0 Oct 30 19:58:20.169253 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.169191 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerDied","Data":"e2cea8dcf26cb1fa948d7853efd3e96147011ca9f3cb52b303f37ffcc13234d6"} Oct 30 19:58:20.169645 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.169621 4524 scope.go:122] "RemoveContainer" containerID="e2cea8dcf26cb1fa948d7853efd3e96147011ca9f3cb52b303f37ffcc13234d6" Oct 30 19:58:20.171021 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.170996 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" event={"ID":"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1","Type":"ContainerStarted","Data":"4e3f7043e69ec25c180b9ad9e7e4668fe25f6eab1ceaf987665c5fe42ad2842f"} Oct 30 19:58:20.171166 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.171143 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:20.171217 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.171209 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:20.171252 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:20.171241 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:21.181128 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.181036 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" event={"ID":"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1","Type":"ContainerStarted","Data":"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c"} Oct 30 19:58:21.181612 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.181173 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:21.182795 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.182764 4524 generic.go:356] "Generic (PLEG): container finished" podID="280d856f-7580-44f5-a96e-f3fbc4355780" containerID="66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3" exitCode=0 Oct 30 19:58:21.182926 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.182827 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerDied","Data":"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3"} Oct 30 19:58:21.183225 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.183204 4524 scope.go:122] "RemoveContainer" containerID="66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3" Oct 30 19:58:21.184547 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.184508 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:21.184629 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.184581 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:21.184629 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.184622 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:21.184774 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.184757 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerStarted","Data":"517ff5474451a592581b385f3b44b739781b900191f36818c58ce5568863f15a"} Oct 30 19:58:21.302159 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.302094 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" podStartSLOduration=2.5323699079999997 podStartE2EDuration="3.302080507s" totalImagesPullingTime="769.710599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:21.299832587 +0000 UTC m=+996.143504217" watchObservedRunningTime="2026-10-30 19:58:21.302080507 +0000 UTC m=+996.145752164" Oct 30 19:58:21.466776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.453044 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:58:21.466776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.453159 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:58:21.466776 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:21.453199 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:58:22.209740 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:22.208331 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerStarted","Data":"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f"} Oct 30 19:58:22.453722 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:22.452118 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:58:22.453722 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:22.452181 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:58:22.590587 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:22.590496 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.803084 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:22.803042 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.166721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:23.164044 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:58:23.166721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:23.164160 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:58:23.166721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:23.164203 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:58:23.450049 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:23.449967 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:58:23.450626 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:23.450596 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:58:24.008258 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.008226 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 19:58:24.010799 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.010780 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.066634 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.066605 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-f75272-serving-cert\"" Oct 30 19:58:24.102322 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.102270 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.102508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.102369 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prprl\" (UniqueName: \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.102508 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.102408 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.102631 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.102523 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.118161 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.118129 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 19:58:24.170281 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.170247 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:58:24.170552 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.170524 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:58:24.204720 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.203263 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.205843 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.205819 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.206464 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.206444 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prprl\" (UniqueName: \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.206990 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.206972 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.207336 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.205667 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.207466 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:24.207174 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-serving-cert: secret "maas-api-e2e-worker-w6-f75272-serving-cert" not found Oct 30 19:58:24.207631 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:24.207621 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls podName:ee87cffe-c28e-4278-9d4a-087968dd6f4c nodeName:}" failed. No retries permitted until 2026-10-30 19:58:24.707600396 +0000 UTC m=+999.551272019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls") pod "maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" (UID: "ee87cffe-c28e-4278-9d4a-087968dd6f4c") : secret "maas-api-e2e-worker-w6-f75272-serving-cert" not found Oct 30 19:58:24.211402 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.211380 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.302732 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.302632 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.315744 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.313380 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prprl\" (UniqueName: \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.450726 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.449963 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:58:24.450726 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.450018 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:58:24.521060 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.520810 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.721820 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.721782 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.724207 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.724184 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:24.871733 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.868264 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.923578 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:24.923541 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:25.175723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:25.175503 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:58:25.175723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:25.175566 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:58:25.341352 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:25.341327 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 19:58:25.449992 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:25.449957 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:58:25.450271 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:25.450243 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:58:26.174926 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.174898 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:58:26.175272 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.174953 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:58:26.233344 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.233316 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" event={"ID":"ee87cffe-c28e-4278-9d4a-087968dd6f4c","Type":"ContainerStarted","Data":"8cf2b596083fe7f448bedb5fa16351a85412e3f8715b186fa7e245d34760ff60"} Oct 30 19:58:26.472416 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.472332 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:58:26.472761 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.472739 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:58:26.473713 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:26.473681 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 19:58:27.165102 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.165063 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:58:27.165389 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.165359 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:58:27.214048 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.214024 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 19:58:27.240228 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.240194 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" event={"ID":"ee87cffe-c28e-4278-9d4a-087968dd6f4c","Type":"ContainerStarted","Data":"1c6a808db9a418969d4b7b5d20ce8765de244b95b3668980c4d502702c48c42e"} Oct 30 19:58:27.240391 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.240377 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:27.452681 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:27.452571 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" podStartSLOduration=3.662814007 podStartE2EDuration="4.452555151s" totalImagesPullingTime="789.741144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:27.451388921 +0000 UTC m=+1002.295060551" watchObservedRunningTime="2026-10-30 19:58:27.452555151 +0000 UTC m=+1002.296226780" Oct 30 19:58:28.167303 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:28.167275 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:58:28.167689 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:28.167667 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:58:28.168397 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:28.168378 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 19:58:29.380256 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:29.380207 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.590930 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:29.590898 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.796994 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:29.796964 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.251513 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:33.251485 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 19:58:34.396954 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:34.396923 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.474690 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:38.474660 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.692472 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:38.692433 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.535868 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:39.535836 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.931540 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:39.931499 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.772655 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:42.772590 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.795649 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:58:42.795617 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:58:43.417822 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:43.417785 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.639985 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:43.639944 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.131435 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:45.129522 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.761206 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:46.761159 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.965884 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:46.965841 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:58:46.966055 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:46.965927 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:48.056891 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:48.056852 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.737942 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:48.737912 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.993937 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:48.993859 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.355961 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.355868 4524 generic.go:356] "Generic (PLEG): container finished" podID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerID="45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc" exitCode=0 Oct 30 19:58:49.355961 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.355935 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerDied","Data":"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc"} Oct 30 19:58:49.356405 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.356289 4524 scope.go:122] "RemoveContainer" containerID="45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc" Oct 30 19:58:49.357595 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.357565 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:49.357721 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.357648 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:49.357796 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:49.357695 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:50.374592 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:50.374555 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerStarted","Data":"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6"} Oct 30 19:58:50.814539 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:50.814502 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:50.865724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:50.865058 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:58:50.865724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:50.865215 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:58:50.865724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:50.865261 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:58:51.865723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:51.865025 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:58:51.865723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:51.865088 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:58:52.866723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:52.866059 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:58:52.866723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:52.866121 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:58:53.343129 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:53.343092 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.867723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:53.867257 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:58:53.867723 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:53.867330 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:58:53.971302 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:53.971264 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.269724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:54.269284 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.865243 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:54.865206 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:58:54.865526 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:54.865495 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:58:55.871768 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:55.871735 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:58:55.872273 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:55.872093 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:58:55.872724 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:55.872680 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 19:58:56.154726 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:56.154661 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.851420 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:58.851382 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.476436 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:59.476400 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.705770 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:58:59.705736 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:01.461296 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:01.461258 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.639379 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:06.639348 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:08.410480 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:08.410431 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.338958 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:09.338921 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.988138 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:09.988092 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.369077 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:10.369011 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.673925 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:11.673893 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.785651 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:59:12.785614 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:59:13.436662 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:13.436631 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.098104 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:14.098067 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.314752 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:14.314687 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:15.935325 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:15.935294 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.775343 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:16.775308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.880061 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:19.880026 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.530333 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:20.530303 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.827428 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:20.827346 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.047306 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:22.047266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.575383 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:23.575335 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:26.936320 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:26.936290 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.653202 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:31.653158 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.571793 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:36.571740 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.415576 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:38.415241 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.171596 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:40.171561 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:41.402557 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:41.402516 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.783780 ip-10-0-130-41 kubenswrapper[4524]: E1030 19:59:42.783745 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 19:59:46.291953 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:46.291903 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.194312 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:51.194275 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.687077 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:51.687015 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.400465 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:55.400372 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.920067 ip-10-0-130-41 kubenswrapper[4524]: I1030 19:59:55.920028 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.764168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:00.764133 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:01.623594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:01.623556 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.630039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:05.630000 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.631119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:10.631069 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.787958 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:00:12.787920 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:00:15.446613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:15.446578 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.035261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.035228 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:00:16.038595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.038573 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.091650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.091625 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"default-dockercfg-wdcjk\"" Oct 30 20:00:16.110287 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.110263 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"premium-w0-779921-kserve-self-signed-certs\"" Oct 30 20:00:16.142143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.142120 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"openshift-service-ca.crt\"" Oct 30 20:00:16.142328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.142247 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"kube-root-ca.crt\"" Oct 30 20:00:16.144261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.144239 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:00:16.146709 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.146679 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.146818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.146794 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.146869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.146854 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.146953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.146941 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.147047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.147029 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h9p\" (UniqueName: \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.147105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.147063 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.192144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.192090 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.247907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.247870 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h9p\" (UniqueName: \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.247911 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.247986 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248018 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248041 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248191 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248487 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248530 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.248675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.248646 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.250317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.250292 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.250658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.250639 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.312685 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.312628 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h9p\" (UniqueName: \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\") pod \"premium-w0-779921-kserve-9f646dd9-gzrfx\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.351663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.351634 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:16.739566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.739539 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:00:16.740592 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:00:16.740565 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3896eeb3_6acb_424e_954b_9d5e35634d7b.slice/crio-0f6221e2bbc3179c0de836803500492c57ba78880833017c2de316b9c010bed1 WatchSource:0}: Error finding container 0f6221e2bbc3179c0de836803500492c57ba78880833017c2de316b9c010bed1: Status 404 returned error can't find the container with id 0f6221e2bbc3179c0de836803500492c57ba78880833017c2de316b9c010bed1 Oct 30 20:00:16.829183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.829151 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" event={"ID":"3896eeb3-6acb-424e-954b-9d5e35634d7b","Type":"ContainerStarted","Data":"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae"} Oct 30 20:00:16.829348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.829196 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" event={"ID":"3896eeb3-6acb-424e-954b-9d5e35634d7b","Type":"ContainerStarted","Data":"0f6221e2bbc3179c0de836803500492c57ba78880833017c2de316b9c010bed1"} Oct 30 20:00:16.948631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:16.948572 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podStartSLOduration=1.948558376 podStartE2EDuration="1.948558376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:16.946501319 +0000 UTC m=+1111.790172995" watchObservedRunningTime="2026-10-30 20:00:16.948558376 +0000 UTC m=+1111.792230006" Oct 30 20:00:17.352622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:17.352585 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:20.429599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:20.429555 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:25.478965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:25.478926 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.352305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:26.352275 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:26.365054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:26.365025 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:26.893550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:26.893521 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:00:30.378278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:30.378229 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.834136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:32.834097 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.274122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:35.274083 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.103318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:40.103276 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.523865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:40.523826 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.785213 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:00:42.785179 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:00:44.080094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.080063 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8lr4s-xxkwp"] Oct 30 20:00:44.083775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.083757 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.145189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.145137 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.145346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.145209 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.145346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.145261 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.145472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.145352 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwn6\" (UniqueName: \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.188930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.188895 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hpq6h\"" Oct 30 20:00:44.189081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.188997 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 20:00:44.190993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.190969 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8lr4s-xxkwp"] Oct 30 20:00:44.246382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.246352 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.246382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.246384 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.246562 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.246401 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.246562 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.246463 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwn6\" (UniqueName: \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.246757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.246739 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.247073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.247056 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.290310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.290279 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 20:00:44.298004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.297985 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.308833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.308810 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwn6\" (UniqueName: \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\") pod \"periodic-gathering-8lr4s-xxkwp\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.396329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.396304 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:00:44.769443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.769416 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8lr4s-xxkwp"] Oct 30 20:00:44.772289 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:00:44.772261 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f714b5_a587_48ce_a229_8487115a742a.slice/crio-441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a WatchSource:0}: Error finding container 441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a: Status 404 returned error can't find the container with id 441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a Oct 30 20:00:44.863575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.863515 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.981315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.981280 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" event={"ID":"c6f714b5-a587-48ce-a229-8487115a742a","Type":"ContainerStarted","Data":"96d0b9cb760635787d4bac0f8b377505a13ab90b287d995f14879abbebc3359b"} Oct 30 20:00:44.981485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:44.981334 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" event={"ID":"c6f714b5-a587-48ce-a229-8487115a742a","Type":"ContainerStarted","Data":"441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a"} Oct 30 20:00:45.099797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:45.099683 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" podStartSLOduration=1.099669963 podStartE2EDuration="1.099669963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:45.098074167 +0000 UTC m=+1139.941745800" watchObservedRunningTime="2026-10-30 20:00:45.099669963 +0000 UTC m=+1139.943341591" Oct 30 20:00:49.809375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:49.809336 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.635378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:54.635309 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:58.420285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:58.420247 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.586607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:00:59.586577 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.558359 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:04.558318 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.607607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:08.607568 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.491292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:09.491258 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.786471 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:01:12.786425 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:01:14.484055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:14.484024 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.467974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:19.467936 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.646271 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:21.646236 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:24.425391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:24.425354 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.307586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:29.307547 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.157540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.157506 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:01:33.162911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.162886 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.220346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.220316 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:01:33.267364 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.267336 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"default-dockercfg-h8x5s\"" Oct 30 20:01:33.267541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.267397 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"kube-root-ca.crt\"" Oct 30 20:01:33.267541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.267456 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"openshift-service-ca.crt\"" Oct 30 20:01:33.267541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.267494 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"premium-w4-bebb42-kserve-self-signed-certs\"" Oct 30 20:01:33.281986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.281965 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.282099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.281995 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwql2\" (UniqueName: \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.282099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.282026 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.282230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.282209 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.282288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.282260 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.282405 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.282387 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.327283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.327249 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.383344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383314 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383355 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383417 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383441 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383579 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwql2\" (UniqueName: \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383626 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383832 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383883 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.383911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.383900 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.385775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.385749 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.386124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.386103 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.445911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.445830 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwql2\" (UniqueName: \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\") pod \"premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.474728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.474686 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:33.852298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:33.852195 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:01:33.855299 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:01:33.855270 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4801444_2aa3_4e3b_946e_e1dea3aa32c6.slice/crio-8ca4a42bdd6465e81a001b1d8b9615f36c5f98f5e890ca818845a218bfbf43da WatchSource:0}: Error finding container 8ca4a42bdd6465e81a001b1d8b9615f36c5f98f5e890ca818845a218bfbf43da: Status 404 returned error can't find the container with id 8ca4a42bdd6465e81a001b1d8b9615f36c5f98f5e890ca818845a218bfbf43da Oct 30 20:01:34.127454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:34.127331 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.253865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:34.253828 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" event={"ID":"c4801444-2aa3-4e3b-946e-e1dea3aa32c6","Type":"ContainerStarted","Data":"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4"} Oct 30 20:01:34.253865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:34.253872 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" event={"ID":"c4801444-2aa3-4e3b-946e-e1dea3aa32c6","Type":"ContainerStarted","Data":"8ca4a42bdd6465e81a001b1d8b9615f36c5f98f5e890ca818845a218bfbf43da"} Oct 30 20:01:34.405961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:34.405903 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podStartSLOduration=1.405884939 podStartE2EDuration="1.405884939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:34.403750703 +0000 UTC m=+1189.247422333" watchObservedRunningTime="2026-10-30 20:01:34.405884939 +0000 UTC m=+1189.249556570" Oct 30 20:01:34.475435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:34.475410 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:39.334562 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:39.334523 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.889813 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:40.889743 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.787843 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:01:42.787807 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:01:43.475021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:43.474985 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:43.487505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:43.487474 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:44.269805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.269775 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.324956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.324924 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:01:44.580780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.580665 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:01:44.585304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.585280 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608165 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608468 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608206 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nz5\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608468 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608229 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608468 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608326 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608468 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608430 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608468 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608460 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608491 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608564 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.608839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.608633 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.643419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.643391 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f84d85-openshift-default-dockercfg-lv5m9\"" Oct 30 20:01:44.650483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.650448 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:01:44.709515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709476 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.709775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709566 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.709775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709599 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.709775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709639 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.709775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709672 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709851 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709884 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nz5\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.709915 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.710023 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.710598 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.710650 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.710921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.710892 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.711032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.710691 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.712648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.712625 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.712857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.712838 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.772041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.772012 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.772181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.772085 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nz5\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:44.899459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:44.899433 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:45.238808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.238733 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:01:45.243420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.243395 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.299210 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.299174 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d67ea9-openshift-default-dockercfg-t5857\"" Oct 30 20:01:45.309159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.309122 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315665 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315734 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315784 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315851 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315888 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315912 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315931 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315968 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.317731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.315994 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wpr9\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.338104 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:01:45.338080 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c00b976_7d95_4eb0_b308_0dc214523fc3.slice/crio-4936ca8abca9230786816127b39d709fa3a941539eda788b1a541bbc29d709dd WatchSource:0}: Error finding container 4936ca8abca9230786816127b39d709fa3a941539eda788b1a541bbc29d709dd: Status 404 returned error can't find the container with id 4936ca8abca9230786816127b39d709fa3a941539eda788b1a541bbc29d709dd Oct 30 20:01:45.340310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.340279 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:45.340411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.340358 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:45.340411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.340401 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:45.350908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.350882 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:01:45.416773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416748 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wpr9\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.416869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416805 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.416869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416837 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.416974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416874 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.416974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416924 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.416974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416958 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.416981 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417000 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417033 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417351 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417545 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417815 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417835 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.417914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.417816 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.419611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.419590 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.419719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.419596 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.485206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.485173 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wpr9\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.486269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.486246 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.560162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.560076 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:45.945298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.942150 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:01:45.946420 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:01:45.946390 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe1cfe_720c_4c29_b5d9_f2b109d689f1.slice/crio-e8c914d05627bf08482f5126f6fbb1f85935a5b249719db2bd8462a5f308da69 WatchSource:0}: Error finding container e8c914d05627bf08482f5126f6fbb1f85935a5b249719db2bd8462a5f308da69: Status 404 returned error can't find the container with id e8c914d05627bf08482f5126f6fbb1f85935a5b249719db2bd8462a5f308da69 Oct 30 20:01:45.948925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.948890 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:45.949038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.948973 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:45.949038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:45.949020 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:46.327497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.327397 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" event={"ID":"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1","Type":"ContainerStarted","Data":"d93002bc151f415f3e6dcb6b524433fd6a9a7d35030d1aeadf9deb96c43ae977"} Oct 30 20:01:46.327497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.327461 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" event={"ID":"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1","Type":"ContainerStarted","Data":"e8c914d05627bf08482f5126f6fbb1f85935a5b249719db2bd8462a5f308da69"} Oct 30 20:01:46.329006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.328982 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" event={"ID":"2c00b976-7d95-4eb0-b308-0dc214523fc3","Type":"ContainerStarted","Data":"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9"} Oct 30 20:01:46.329139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.329022 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" event={"ID":"2c00b976-7d95-4eb0-b308-0dc214523fc3","Type":"ContainerStarted","Data":"4936ca8abca9230786816127b39d709fa3a941539eda788b1a541bbc29d709dd"} Oct 30 20:01:46.477721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.475680 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podStartSLOduration=1.475662468 podStartE2EDuration="1.475662468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:46.475049251 +0000 UTC m=+1201.318720881" watchObservedRunningTime="2026-10-30 20:01:46.475662468 +0000 UTC m=+1201.319334097" Oct 30 20:01:46.565722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.564950 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:46.600138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.600019 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podStartSLOduration=2.599999845 podStartE2EDuration="2.599999845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:46.597216071 +0000 UTC m=+1201.440887700" watchObservedRunningTime="2026-10-30 20:01:46.599999845 +0000 UTC m=+1201.443671475" Oct 30 20:01:46.901567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.901527 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:46.909201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.908827 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:46.909201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:46.908888 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:47.571721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:47.571445 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:47.571721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:47.571507 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:47.902747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:47.900092 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:47.902747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:47.900152 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:48.586724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:48.568120 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:48.586724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:48.568194 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:48.902015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:48.901330 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:48.902015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:48.901426 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:49.574721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:49.574083 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:49.574721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:49.574198 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:49.596724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:49.589158 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.899990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:49.899961 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:49.900228 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:49.900202 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:50.567725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:50.567035 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:50.567725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:50.567100 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:50.899967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:50.899927 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:50.900477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:50.899979 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:51.569041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:51.569006 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:51.569299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:51.569269 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:51.900476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:51.900443 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 20:01:51.900958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:51.900500 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 20:01:52.560999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:52.560948 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:01:52.561184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:52.561030 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:01:52.904943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:52.904912 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:52.905434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:52.905270 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:52.906089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:52.906065 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:01:53.564731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:53.564685 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:53.565006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:53.564988 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:53.565794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:53.565778 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:01:54.631155 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:54.631124 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.769634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:01:59.769534 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.510713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:04.510665 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.656142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:04.656107 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 20:02:04.656369 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:04.656343 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" containerID="cri-o://df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072" gracePeriod=30 Oct 30 20:02:07.648252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:07.648221 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.394876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.394839 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.910462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.910437 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 20:02:09.980808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980745 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.980808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980794 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.980964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980862 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.980964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980917 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.980964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980949 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.981118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.980988 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.981118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981036 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.981219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981138 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.981219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981170 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\") pod \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\" (UID: \"bbf7171d-7aef-46b7-9af6-d3e0773a41b2\") " Oct 30 20:02:09.981327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981136 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:09.981327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981262 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:09.981327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981246 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:09.981487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981410 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:09.981631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981610 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:09.981739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981641 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:09.981739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981657 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:09.981739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981672 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:09.981739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.981615 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:09.982903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.982874 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:09.983179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.983156 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "kube-api-access-rxp4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:09.983179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.983168 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:09.983481 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:09.983459 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy" pod "bbf7171d-7aef-46b7-9af6-d3e0773a41b2" (UID: "bbf7171d-7aef-46b7-9af6-d3e0773a41b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:10.082263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.082235 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxp4k\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-kube-api-access-rxp4k\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.082372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.082266 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.082372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.082281 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.082372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.082294 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.082372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.082308 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf7171d-7aef-46b7-9af6-d3e0773a41b2-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.516769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.516730 4524 generic.go:356] "Generic (PLEG): container finished" podID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerID="df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072" exitCode=0 Oct 30 20:02:10.517207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.516771 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" event={"ID":"bbf7171d-7aef-46b7-9af6-d3e0773a41b2","Type":"ContainerDied","Data":"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072"} Oct 30 20:02:10.517207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.516841 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" event={"ID":"bbf7171d-7aef-46b7-9af6-d3e0773a41b2","Type":"ContainerDied","Data":"cc7511eb33e00ccee698927772d146727a7a22bcee5151fea0d6151fd3bea605"} Oct 30 20:02:10.517207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.516780 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs" Oct 30 20:02:10.517207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.516870 4524 scope.go:122] "RemoveContainer" containerID="df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072" Oct 30 20:02:10.526309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.526292 4524 scope.go:122] "RemoveContainer" containerID="df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072" Oct 30 20:02:10.526601 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:02:10.526579 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072\": container with ID starting with df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072 not found: ID does not exist" containerID="df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072" Oct 30 20:02:10.526639 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.526614 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072"} err="failed to get container status \"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072\": rpc error: code = NotFound desc = could not find container \"df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072\": container with ID starting with df027d2758352b1de09c066a9ab1d109e0c754aafe4013fa6ab6932481337072 not found: ID does not exist" Oct 30 20:02:10.683721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.683653 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 20:02:10.734869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:10.734828 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2ljcs"] Oct 30 20:02:11.299485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:11.299451 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" path="/var/lib/kubelet/pods/bbf7171d-7aef-46b7-9af6-d3e0773a41b2/volumes" Oct 30 20:02:12.786791 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:02:12.786750 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:02:14.261182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:14.261147 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.202269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:19.202185 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.603089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:21.602586 4524 generic.go:356] "Generic (PLEG): container finished" podID="c6f714b5-a587-48ce-a229-8487115a742a" containerID="96d0b9cb760635787d4bac0f8b377505a13ab90b287d995f14879abbebc3359b" exitCode=0 Oct 30 20:02:21.603089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:21.602667 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" event={"ID":"c6f714b5-a587-48ce-a229-8487115a742a","Type":"ContainerDied","Data":"96d0b9cb760635787d4bac0f8b377505a13ab90b287d995f14879abbebc3359b"} Oct 30 20:02:22.777370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.777346 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:02:22.847968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.847934 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\") pod \"c6f714b5-a587-48ce-a229-8487115a742a\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " Oct 30 20:02:22.848147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848034 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\") pod \"c6f714b5-a587-48ce-a229-8487115a742a\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " Oct 30 20:02:22.848147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848069 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\") pod \"c6f714b5-a587-48ce-a229-8487115a742a\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " Oct 30 20:02:22.848147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848144 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\" (UniqueName: \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\") pod \"c6f714b5-a587-48ce-a229-8487115a742a\" (UID: \"c6f714b5-a587-48ce-a229-8487115a742a\") " Oct 30 20:02:22.848399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848347 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle" pod "c6f714b5-a587-48ce-a229-8487115a742a" (UID: "c6f714b5-a587-48ce-a229-8487115a742a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:22.848497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848448 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path" pod "c6f714b5-a587-48ce-a229-8487115a742a" (UID: "c6f714b5-a587-48ce-a229-8487115a742a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:22.848550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.848487 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle" pod "c6f714b5-a587-48ce-a229-8487115a742a" (UID: "c6f714b5-a587-48ce-a229-8487115a742a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:22.850485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.850461 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6" pod "c6f714b5-a587-48ce-a229-8487115a742a" (UID: "c6f714b5-a587-48ce-a229-8487115a742a"). InnerVolumeSpecName "kube-api-access-5lwn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:22.949852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.949821 4524 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-trusted-ca-bundle\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:22.949852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.949848 4524 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f714b5-a587-48ce-a229-8487115a742a-service-ca-bundle\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:22.950081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.949861 4524 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f714b5-a587-48ce-a229-8487115a742a-archives-path\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:22.950081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:22.949872 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwn6\" (UniqueName: \"kubernetes.io/projected/c6f714b5-a587-48ce-a229-8487115a742a-kube-api-access-5lwn6\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:23.614999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:23.614922 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" Oct 30 20:02:23.615149 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:23.614920 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8lr4s-xxkwp" event={"ID":"c6f714b5-a587-48ce-a229-8487115a742a","Type":"ContainerDied","Data":"441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a"} Oct 30 20:02:23.615149 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:23.615051 4524 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441020f2bf875a4d973fec8a1b681c9e24f426ec0d7b530fdf9a46a61917ee2a" Oct 30 20:02:24.124552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:24.124515 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:24.666980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:24.666925 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:24.890964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:24.890923 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.103340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:29.103302 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.901269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:33.901224 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.894370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894342 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:02:34.894841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894826 4524 cpu_manager.go:414] "containerMap: removing container" podUID="c6f714b5-a587-48ce-a229-8487115a742a" containerName="insights-gathering" Oct 30 20:02:34.894890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894843 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f714b5-a587-48ce-a229-8487115a742a" containerName="insights-gathering" Oct 30 20:02:34.894890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894853 4524 cpu_manager.go:414] "containerMap: removing container" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" Oct 30 20:02:34.894890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894858 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" Oct 30 20:02:34.894984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894948 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f714b5-a587-48ce-a229-8487115a742a" containerName="insights-gathering" Oct 30 20:02:34.894984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.894957 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf7171d-7aef-46b7-9af6-d3e0773a41b2" containerName="istio-proxy" Oct 30 20:02:34.898235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:34.898212 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.004746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.004684 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:02:35.076301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076267 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076315 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076342 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076474 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076505 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076612 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076631 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcvh\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076661 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.076719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.076678 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.177866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.177796 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.177866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.177843 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.177872 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.177946 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.177973 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178047 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178074 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcvh\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178110 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178146 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178482 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178661 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178810 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178845 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.178939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.178818 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.180753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.180723 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.180866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.180774 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.238876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.238858 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.242915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.242893 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcvh\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tn267\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.512214 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.512137 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:35.950725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.950684 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:02:35.952898 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:02:35.952861 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c515898_62e1_4467_9de1_b2aa3a1e627b.slice/crio-77d2a0a51236b3113827933888392fbc9e935661a58ee34aa4325bed282ec2c6 WatchSource:0}: Error finding container 77d2a0a51236b3113827933888392fbc9e935661a58ee34aa4325bed282ec2c6: Status 404 returned error can't find the container with id 77d2a0a51236b3113827933888392fbc9e935661a58ee34aa4325bed282ec2c6 Oct 30 20:02:35.955453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.955412 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:35.955548 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.955527 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:35.955589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:35.955575 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:36.684939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:36.684900 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" event={"ID":"8c515898-62e1-4467-9de1-b2aa3a1e627b","Type":"ContainerStarted","Data":"37f6a7665d372fd749a5a528dc5fb88e7f94c1b2e56bbaf467b8540181cab9c4"} Oct 30 20:02:36.685409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:36.684960 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" event={"ID":"8c515898-62e1-4467-9de1-b2aa3a1e627b","Type":"ContainerStarted","Data":"77d2a0a51236b3113827933888392fbc9e935661a58ee34aa4325bed282ec2c6"} Oct 30 20:02:36.863720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:36.862845 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podStartSLOduration=2.862823628 podStartE2EDuration="2.862823628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:36.862072799 +0000 UTC m=+1251.705744428" watchObservedRunningTime="2026-10-30 20:02:36.862823628 +0000 UTC m=+1251.706495272" Oct 30 20:02:37.512428 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:37.512385 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:37.514095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:37.514065 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:37.514277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:37.514134 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:38.512622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:38.512588 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:38.513051 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:38.512651 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:39.335959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:39.335924 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.516119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:39.516033 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:39.540943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:39.540900 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:39.644416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:39.644388 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.512598 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:40.512563 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:40.512780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:40.512633 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:41.273108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:41.273071 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.526187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:41.526063 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:41.526187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:41.526153 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:42.515244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:42.515196 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:42.515920 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:42.515882 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:42.886094 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:02:42.885986 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:02:43.512607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:43.512559 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tn267 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:02:43.512884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:43.512638 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 20:02:44.385293 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.385256 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:02:44.389212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.389193 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.444016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.443989 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f75272-openshift-default-dockercfg-9blkz\"" Oct 30 20:02:44.473303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.473274 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.496090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.496064 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:02:44.510955 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.510932 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.510998 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511056 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511138 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511187 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511248 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511297 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511319 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7n6\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.511408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.511394 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.612594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612554 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.612807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612610 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.612807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612641 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.612807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612666 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.612807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612694 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.612985 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613032 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7n6\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613111 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613128 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613200 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613393 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613453 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613745 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.613959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.613935 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.615473 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.615447 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.615829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.615809 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.687936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.687896 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.688181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.688153 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7n6\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.702854 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.702825 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:44.994483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.994396 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:44.994843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.994814 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:44.995607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:44.995583 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:02:45.159235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.159205 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:02:45.162061 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:02:45.162031 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c2a6ab_82ca_4b50_98d0_7d25ca8b3641.slice/crio-e8f3f8359fb6f359cc2ae070740c71095df8b474e672f95ae82ff5e2178d6574 WatchSource:0}: Error finding container e8f3f8359fb6f359cc2ae070740c71095df8b474e672f95ae82ff5e2178d6574: Status 404 returned error can't find the container with id e8f3f8359fb6f359cc2ae070740c71095df8b474e672f95ae82ff5e2178d6574 Oct 30 20:02:45.164857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.164827 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:45.164968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.164900 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:45.165086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.164971 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:45.749725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.749607 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" event={"ID":"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641","Type":"ContainerStarted","Data":"61c62cac9a92d205b3d83ccbd7292812ba5199ef90aa15bcffa78828c7fcf98a"} Oct 30 20:02:45.750216 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:45.749734 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" event={"ID":"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641","Type":"ContainerStarted","Data":"e8f3f8359fb6f359cc2ae070740c71095df8b474e672f95ae82ff5e2178d6574"} Oct 30 20:02:46.704358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:46.704244 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:46.705791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:46.705756 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:46.706027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:46.705998 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:47.704719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:47.704244 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:47.704719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:47.704312 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:48.703969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:48.703929 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:48.704144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:48.703991 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:49.575587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:49.575542 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.704190 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:49.704156 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:49.704361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:49.704219 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:50.025785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.025725 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podStartSLOduration=6.02569157 podStartE2EDuration="6.02569157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:45.887837182 +0000 UTC m=+1260.731508814" watchObservedRunningTime="2026-10-30 20:02:50.02569157 +0000 UTC m=+1264.869363198" Oct 30 20:02:50.026215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.026190 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:02:50.031341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.031322 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.077131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.077095 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:02:50.080681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.080661 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.125650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.125618 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:02:50.195897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.195845 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:02:50.201495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201467 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.201647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201505 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.201647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201529 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.201751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201677 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdpv\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.201751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201736 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.201829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201767 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.201829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201791 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcnk\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.201829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201820 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.201969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201842 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.201969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201880 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.201969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.201899 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.202120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202080 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.202178 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202113 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.202178 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202148 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.202278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202213 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.202330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202277 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.202330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202316 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.202427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.202346 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.302935 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.302988 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303014 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303033 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303053 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303072 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303107 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303255 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdpv\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303313 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303352 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303388 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcnk\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303402 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303432 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303466 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303469 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303480 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.303551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303537 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303572 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303614 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303641 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303669 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303781 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303805 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303857 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303875 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.304032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.303876 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.304313 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.304501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.304387 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.305954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.305923 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.306080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.306010 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.306080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.306068 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.306205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.306162 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.372042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.371988 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.372187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.372154 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdpv\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.372719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.372679 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8pkrs\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.372781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.372721 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcnk\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fpj82\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.395683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.395659 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:50.644886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.644856 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:50.703994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.703962 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:50.704121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.704024 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:50.873972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.873929 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:02:50.876912 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:02:50.876887 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59604139_4a1e_49a7_bdeb_53825ca1fb7a.slice/crio-a63c044a7cfd62e10f4898ad3a2785f051a5fe479f811a13904e4c7a8521f3a4 WatchSource:0}: Error finding container a63c044a7cfd62e10f4898ad3a2785f051a5fe479f811a13904e4c7a8521f3a4: Status 404 returned error can't find the container with id a63c044a7cfd62e10f4898ad3a2785f051a5fe479f811a13904e4c7a8521f3a4 Oct 30 20:02:50.879366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.879333 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:50.879446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.879401 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:50.879446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:50.879439 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:51.059654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.059618 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:02:51.080043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.078397 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:51.080043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.078486 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:51.080043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.078523 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:51.704218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.704180 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:02:51.704812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.704234 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 20:02:51.826483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.826450 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" event={"ID":"59604139-4a1e-49a7-bdeb-53825ca1fb7a","Type":"ContainerStarted","Data":"efd7606689ebb05f266c8fd5ce93a5ad1ee5588eca3a0dca9c278f6ece50790b"} Oct 30 20:02:51.826661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.826515 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" event={"ID":"59604139-4a1e-49a7-bdeb-53825ca1fb7a","Type":"ContainerStarted","Data":"a63c044a7cfd62e10f4898ad3a2785f051a5fe479f811a13904e4c7a8521f3a4"} Oct 30 20:02:51.828154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.828129 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" event={"ID":"1924a830-7a78-47f0-a825-37e523111f58","Type":"ContainerStarted","Data":"c035571b1d30b92d5a18aff75abaf74fdbb1f4d6c26a365954fcf03e8b687bb8"} Oct 30 20:02:51.828259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.828168 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" event={"ID":"1924a830-7a78-47f0-a825-37e523111f58","Type":"ContainerStarted","Data":"3ed0babbdd80255e5d98ab6cf0c57af715320adc2a2b2f73e8100b408857db6b"} Oct 30 20:02:51.962781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:51.961664 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podStartSLOduration=1.961643936 podStartE2EDuration="1.961643936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:51.95874914 +0000 UTC m=+1266.802420770" watchObservedRunningTime="2026-10-30 20:02:51.961643936 +0000 UTC m=+1266.805315569" Oct 30 20:02:52.401568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.401536 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:52.402024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.401999 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:52.402220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.402190 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:52.603732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.603232 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.645474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.645442 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:02:52.677788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.677510 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:52.677988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.677679 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:52.708675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.708639 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:52.709077 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.708999 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:52.709687 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.709672 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:02:52.837910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.837862 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podStartSLOduration=3.8378498519999997 podStartE2EDuration="3.837849852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:52.105088913 +0000 UTC m=+1266.948760554" watchObservedRunningTime="2026-10-30 20:02:52.837849852 +0000 UTC m=+1267.681521481" Oct 30 20:02:52.839010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:52.838986 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.396140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:53.396100 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:53.396420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:53.396391 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:53.645465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:53.645432 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:53.645645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:53.645482 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:54.407724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:54.403821 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:54.407724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:54.403887 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:54.649931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:54.649897 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:54.650195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:54.650167 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:54.733062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:54.732971 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.396112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:55.396077 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:55.396374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:55.396341 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:55.645361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:55.645324 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:55.645867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:55.645397 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:56.398076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:56.398032 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:56.398277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:56.398093 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:56.645692 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:56.645657 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:56.646217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:56.645745 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:57.396328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:57.396275 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:57.396514 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:57.396362 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:57.645346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:57.645305 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:57.645516 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:57.645386 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:58.396482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:58.396443 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:02:58.396880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:58.396506 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:02:58.646046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:58.646004 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fpj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:02:58.646218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:58.646071 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 20:02:59.600553 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:59.600517 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:59.601740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:59.601674 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:59.602658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:59.602641 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:02:59.793895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:02:59.793861 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:00.090411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:00.090381 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:03:00.091254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:00.091229 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:03:00.092923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:00.092900 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:03:04.603811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:04.603772 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.438147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:09.438097 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.785872 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:03:12.785828 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:03:14.416255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.416222 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:03:14.420146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.420123 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.473644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.473505 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-407632-openshift-default-dockercfg-5j7qg\"" Oct 30 20:03:14.487226 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487191 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6xp\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487337 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487373 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487485 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487524 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487550 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487650 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487733 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.487787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.487774 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.526014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.525989 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.534202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.534172 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:03:14.588971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.588946 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589002 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589062 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589107 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6xp\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589212 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589251 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589303 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589339 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589374 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589600 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589576 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589759 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589771 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.589961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.589829 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.590052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.590031 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.591695 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.591672 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.591912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.591897 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.650912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.650882 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.651027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.651011 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6xp\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:14.750522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:14.750451 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:15.189910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:15.189885 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:03:15.192048 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:03:15.192017 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4f63b0_12dc_4476_a0a6_b53b975721ee.slice/crio-7a73da447ca612d7cdbfae70cdb5dc349b07bd0c69dcd4d7aca635ff9a2fa79f WatchSource:0}: Error finding container 7a73da447ca612d7cdbfae70cdb5dc349b07bd0c69dcd4d7aca635ff9a2fa79f: Status 404 returned error can't find the container with id 7a73da447ca612d7cdbfae70cdb5dc349b07bd0c69dcd4d7aca635ff9a2fa79f Oct 30 20:03:15.194802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:15.194766 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:15.194880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:15.194828 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:15.194880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:15.194862 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:16.073132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.073085 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" event={"ID":"da4f63b0-12dc-4476-a0a6-b53b975721ee","Type":"ContainerStarted","Data":"935df5f8a573428b015bd860263454994c87d8e14d9aa1c589d9bfe2650a7877"} Oct 30 20:03:16.073792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.073233 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" event={"ID":"da4f63b0-12dc-4476-a0a6-b53b975721ee","Type":"ContainerStarted","Data":"7a73da447ca612d7cdbfae70cdb5dc349b07bd0c69dcd4d7aca635ff9a2fa79f"} Oct 30 20:03:16.245168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.245116 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podStartSLOduration=2.245102989 podStartE2EDuration="2.245102989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:16.244199413 +0000 UTC m=+1291.087871046" watchObservedRunningTime="2026-10-30 20:03:16.245102989 +0000 UTC m=+1291.088774617" Oct 30 20:03:16.762729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.762574 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:16.770746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.766795 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:03:16.770746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:16.766849 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:03:17.767722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:17.761056 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:03:17.767722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:17.761122 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:03:18.759723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:18.757835 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:03:18.759723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:18.757909 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:03:19.396748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:19.383149 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:19.766903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:19.766818 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:03:19.767168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:19.767141 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:03:20.765725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:20.752032 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:03:20.765725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:20.752104 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:03:22.072648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:22.072612 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:22.073563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:22.073533 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:22.074430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:22.074407 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:03:24.191345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:24.191308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.994808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:28.994556 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.216076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:29.216048 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.917656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:33.917623 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.908962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:38.908921 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.127985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:39.127950 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.628031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:41.627994 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.786336 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:03:42.786298 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:03:43.808341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:43.808303 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.645581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:48.645495 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.439366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:53.439331 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.564618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:03:58.564563 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.371946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:03.371903 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.349739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:08.349625 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.785831 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:04:12.785795 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:04:13.172189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:13.172117 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.236240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:18.236196 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.459217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:18.459182 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.583973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:19.583938 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.079391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:23.079353 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:28.057720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:28.050602 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.300793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.300756 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:04:30.304680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.304657 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.409625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.409589 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:04:30.418611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.418582 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.418797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.418651 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.418841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.418803 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.418879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.418868 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.418913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.418893 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxvt\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.419023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.419003 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.419093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.419046 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.419148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.419129 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.419196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.419185 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520671 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520736 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520761 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxvt\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520795 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520818 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.520890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520855 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520919 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.520952 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521052 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521126 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521218 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521272 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521418 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.521805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.521785 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.523441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.523419 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.523544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.523506 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.584927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.584864 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxvt\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.584927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.584916 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:30.617586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:30.617557 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:31.053657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.053632 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:04:31.057122 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:04:31.057089 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd340e01_9c01_48e1_a732_89bbb7440cc0.slice/crio-681cfa3e4313250138074630e0d90f1bbdf68199281aaddcd503aec6afc86509 WatchSource:0}: Error finding container 681cfa3e4313250138074630e0d90f1bbdf68199281aaddcd503aec6afc86509: Status 404 returned error can't find the container with id 681cfa3e4313250138074630e0d90f1bbdf68199281aaddcd503aec6afc86509 Oct 30 20:04:31.059535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.059501 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:31.059656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.059575 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:31.059656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.059611 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:31.526714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.526660 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" event={"ID":"cd340e01-9c01-48e1-a732-89bbb7440cc0","Type":"ContainerStarted","Data":"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666"} Oct 30 20:04:31.527192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.526735 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" event={"ID":"cd340e01-9c01-48e1-a732-89bbb7440cc0","Type":"ContainerStarted","Data":"681cfa3e4313250138074630e0d90f1bbdf68199281aaddcd503aec6afc86509"} Oct 30 20:04:31.617788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.617758 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:31.755727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:31.753898 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podStartSLOduration=1.753877138 podStartE2EDuration="1.753877138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:31.698109408 +0000 UTC m=+1366.541781037" watchObservedRunningTime="2026-10-30 20:04:31.753877138 +0000 UTC m=+1366.597548771" Oct 30 20:04:32.620156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:32.620122 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:32.620550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:32.620179 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:32.965728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:32.961756 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:33.622044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:33.622010 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:33.622671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:33.622638 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:34.634522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:34.625884 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:34.634522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:34.625948 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:35.621720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:35.620174 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:35.621720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:35.620240 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:36.618652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:36.618619 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:36.619069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:36.618676 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:37.618690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:37.618653 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:04:37.619220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:37.618724 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 20:04:38.478212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:38.478174 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.948092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:38.948058 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:38.948810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:38.948741 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:38.950943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:38.950909 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:04:39.514637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:39.502912 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.988047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:41.988017 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.785282 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:04:42.785246 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:04:43.418316 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:43.418276 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.567297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:48.567240 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.364720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:53.364620 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.460031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:04:58.459648 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.281280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.281231 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:05:00.285146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.285125 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.381774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.381741 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.381942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.381797 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.381942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.381893 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.381959 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78vl\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.382024 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.382093 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.382120 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.382233 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.382328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.382311 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.391385 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.391360 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:05:00.483782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.483752 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.483975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.483813 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.483975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.483856 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.483975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.483889 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.483975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.483953 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78vl\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484021 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484071 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484094 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484189 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484292 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484317 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484528 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.484807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.484776 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.485028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.485006 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.486518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.486494 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.486675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.486658 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.545421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.545365 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.545421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.545368 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78vl\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:00.598674 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:00.598652 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:01.038090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.038068 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:05:01.039742 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:01.039692 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8529de96_8528_4d08_add2_5e5644eaa580.slice/crio-baf70453a34517041f632df78ec465f8a655eeb34fa0e4f9c0623c293ab7c47c WatchSource:0}: Error finding container baf70453a34517041f632df78ec465f8a655eeb34fa0e4f9c0623c293ab7c47c: Status 404 returned error can't find the container with id baf70453a34517041f632df78ec465f8a655eeb34fa0e4f9c0623c293ab7c47c Oct 30 20:05:01.042006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.041971 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:01.042153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.042041 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:01.042153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.042072 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:01.558013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.554634 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:05:01.566724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.565970 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603153 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603217 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603246 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dswt\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603313 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603364 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603470 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603524 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603605 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.603641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.603634 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.621672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.621642 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:05:01.704428 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704392 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704448 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704493 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dswt\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704519 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704574 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704606 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704650 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704739 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.704841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.704791 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.705312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.705279 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.705629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.705601 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.705796 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.705747 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.706420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.706241 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.706420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.706378 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.709153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.709129 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.710372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.710350 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.766337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.764791 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" event={"ID":"8529de96-8528-4d08-add2-5e5644eaa580","Type":"ContainerStarted","Data":"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede"} Oct 30 20:05:01.766337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.764903 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" event={"ID":"8529de96-8528-4d08-add2-5e5644eaa580","Type":"ContainerStarted","Data":"baf70453a34517041f632df78ec465f8a655eeb34fa0e4f9c0623c293ab7c47c"} Oct 30 20:05:01.786613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.775805 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dswt\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.786613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.785230 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.834628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.834547 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:05:01.839491 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.839474 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:01.900063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.900025 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:01.902172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.902111 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podStartSLOduration=1.9020920810000002 podStartE2EDuration="1.902092081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:01.900057836 +0000 UTC m=+1396.743729486" watchObservedRunningTime="2026-10-30 20:05:01.902092081 +0000 UTC m=+1396.745763711" Oct 30 20:05:01.907317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:01.907289 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:05:02.008393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008359 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.008527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008423 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.008527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008487 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.008631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008557 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.008631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008587 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.008767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.008685 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hqr\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.010286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.010256 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.010399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.010300 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.010399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.010323 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.111687 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111587 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.111687 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111629 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.111935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111802 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111936 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111972 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.111975 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112039 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112096 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112117 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112347 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112220 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hqr\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112418 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112511 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112462 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112540 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.112687 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.112658 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.114395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.114377 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.114642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.114621 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.179955 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.179879 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.181553 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.181502 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hqr\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.373653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.373620 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:05:02.375670 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:02.375638 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc453eb15_1bf0_48b9_9d46_d2c106f90caf.slice/crio-3b12dfc23ce20fc3d554b7cbf013e769624447bb9aefa4c420cf31abccfc2438 WatchSource:0}: Error finding container 3b12dfc23ce20fc3d554b7cbf013e769624447bb9aefa4c420cf31abccfc2438: Status 404 returned error can't find the container with id 3b12dfc23ce20fc3d554b7cbf013e769624447bb9aefa4c420cf31abccfc2438 Oct 30 20:05:02.378312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.378280 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:02.378393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.378368 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:02.378431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.378399 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:02.481209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.481170 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:02.606645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.606221 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:02.606645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.606328 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:02.606645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.606403 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:02.791007 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.790915 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" event={"ID":"c453eb15-1bf0-48b9-9d46-d2c106f90caf","Type":"ContainerStarted","Data":"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122"} Oct 30 20:05:02.791007 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.790974 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" event={"ID":"c453eb15-1bf0-48b9-9d46-d2c106f90caf","Type":"ContainerStarted","Data":"3b12dfc23ce20fc3d554b7cbf013e769624447bb9aefa4c420cf31abccfc2438"} Oct 30 20:05:02.900931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.900900 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:02.921253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.921191 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podStartSLOduration=1.9211722089999999 podStartE2EDuration="1.921172209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:02.918452531 +0000 UTC m=+1397.762124160" watchObservedRunningTime="2026-10-30 20:05:02.921172209 +0000 UTC m=+1397.764843838" Oct 30 20:05:02.976988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.976952 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:05:02.977503 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:02.977451 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c605388_39c6_4ba7_8938_528e4ed23b91.slice/crio-16dc8f6a05326b4b52b545dcc44860a060a73a73873423b53fe130b11433788b WatchSource:0}: Error finding container 16dc8f6a05326b4b52b545dcc44860a060a73a73873423b53fe130b11433788b: Status 404 returned error can't find the container with id 16dc8f6a05326b4b52b545dcc44860a060a73a73873423b53fe130b11433788b Oct 30 20:05:02.980310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.980273 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:02.980486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.980467 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:02.980635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:02.980619 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:03.416725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.405313 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:03.609732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.602116 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:03.609732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.602190 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:03.817093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.816978 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" event={"ID":"8c605388-39c6-4ba7-8938-528e4ed23b91","Type":"ContainerStarted","Data":"e5ad2e78bc18c46d7f04cc4aa1a053b65a0ba4213b8b9d310ee965d053cdf61d"} Oct 30 20:05:03.817093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.817043 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" event={"ID":"8c605388-39c6-4ba7-8938-528e4ed23b91","Type":"ContainerStarted","Data":"16dc8f6a05326b4b52b545dcc44860a060a73a73873423b53fe130b11433788b"} Oct 30 20:05:03.913725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.913008 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:03.913725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.913076 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:03.943588 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:03.943524 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podStartSLOduration=2.943505053 podStartE2EDuration="2.943505053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.941536097 +0000 UTC m=+1398.785207737" watchObservedRunningTime="2026-10-30 20:05:03.943505053 +0000 UTC m=+1398.787176682" Oct 30 20:05:04.484723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.483872 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:04.497723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.495531 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:04.497723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.495604 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:04.617724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.609086 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:04.617724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.609150 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:04.903493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.903046 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:04.903493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:04.903102 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:05.493733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.493183 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:05.493733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.493245 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:05.601734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.600078 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:05.601734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.600141 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:05.902103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.902065 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:05.903254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:05.902867 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:06.560494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.518051 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:06.560494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.518116 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:06.600201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.600087 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:06.600201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.600145 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:06.937980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.937944 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:06.939423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:06.939039 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:07.497236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.497196 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:07.497433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.497260 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:07.621713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.621288 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:05:07.621713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.621343 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:05:07.918144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.918099 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:07.918408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:07.918381 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:08.485351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.484919 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:08.485351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.484981 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:08.612983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.612955 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:08.613452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.613425 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:08.614177 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.614160 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:05:08.825446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.825319 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.909756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.903143 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:08.909756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:08.903207 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:09.489729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:09.489040 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:05:09.489729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:09.489117 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:05:09.900653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:09.900620 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:05:09.900862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:09.900683 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:05:10.487635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.487595 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:10.487997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.487971 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:10.488872 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.488852 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:05:10.910679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.910643 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:10.913323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.913300 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:10.914411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:10.914370 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:05:12.785114 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:05:12.785069 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:05:13.799722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:13.795058 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.183486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:14.183452 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:16.679238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.679206 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:05:16.683145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.683126 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.790068 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.789726 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:05:16.811238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811204 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811286 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811385 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811435 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811482 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811572 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmnz\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811625 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811733 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.811851 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.811793 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.912937 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.912911 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.912995 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913013 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913060 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913225 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913101 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913225 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913128 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913225 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913150 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913225 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913177 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913230 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmnz\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913341 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913500 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913651 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913633 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.913756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.913737 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.914107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.914089 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.915488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.915466 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.915737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.915714 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.974775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.974674 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.975141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.975118 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmnz\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:16.996300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:16.996277 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:17.430378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:17.430347 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:05:17.431350 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:17.431328 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb20843_de34_4e51_9436_608d9d23950e.slice/crio-ad70d8606f4a8dd66420b6a583523c7f3f1ad6ebe0a80445417854d72e0db4da WatchSource:0}: Error finding container ad70d8606f4a8dd66420b6a583523c7f3f1ad6ebe0a80445417854d72e0db4da: Status 404 returned error can't find the container with id ad70d8606f4a8dd66420b6a583523c7f3f1ad6ebe0a80445417854d72e0db4da Oct 30 20:05:17.434512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:17.434472 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:17.434591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:17.434553 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:17.434642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:17.434594 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:18.105361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.105328 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" event={"ID":"6eb20843-de34-4e51-9436-608d9d23950e","Type":"ContainerStarted","Data":"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3"} Oct 30 20:05:18.105791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.105388 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" event={"ID":"6eb20843-de34-4e51-9436-608d9d23950e","Type":"ContainerStarted","Data":"ad70d8606f4a8dd66420b6a583523c7f3f1ad6ebe0a80445417854d72e0db4da"} Oct 30 20:05:18.283205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.283136 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podStartSLOduration=2.283103961 podStartE2EDuration="2.283103961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:18.282406892 +0000 UTC m=+1413.126078521" watchObservedRunningTime="2026-10-30 20:05:18.283103961 +0000 UTC m=+1413.126775590" Oct 30 20:05:18.964721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.964203 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:18.999721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.999237 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:18.999721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.999367 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:18.999721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:18.999415 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:19.996816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:19.996786 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.997303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:19.996846 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:20.996810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:20.996773 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.996995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:20.996829 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:21.997248 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:21.997209 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:21.997649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:21.997280 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:22.997402 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:22.997364 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:22.997928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:22.997418 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:23.997552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:23.997508 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:05:23.998083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:23.997572 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:05:24.106919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:24.105646 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:25.091137 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:25.091100 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:25.095783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:25.092517 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:25.104303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:25.104266 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:05:26.328351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.328270 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:05:26.332728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.332684 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.385642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.385613 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"kube-root-ca.crt\"" Oct 30 20:05:26.388620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.388597 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"openshift-service-ca.crt\"" Oct 30 20:05:26.438988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.438966 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"default-dockercfg-jjd8c\"" Oct 30 20:05:26.440327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.440301 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:05:26.440642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.440622 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"distinct-w5-407632-kserve-self-signed-certs\"" Oct 30 20:05:26.480443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.480387 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.482472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.482440 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bsj2\" (UniqueName: \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.482719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.482684 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.482952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.482935 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.484171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.484148 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.484407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.484390 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.547693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.547654 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:26.585977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.585899 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.585987 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586024 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586061 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586154 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bsj2\" (UniqueName: \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586185 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586307 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586318 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.586612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.586591 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.588404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.588386 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.588709 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.588679 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.649156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.649124 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bsj2\" (UniqueName: \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\") pod \"distinct-w5-407632-kserve-b79fdcf4d-gv4sx\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:26.945539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:26.945500 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:27.330768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:27.330740 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:05:27.332127 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:27.332102 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6262a9d5_36ff_4999_b1d6_1191b8303485.slice/crio-b56572ac2aff94ee533053895a803de20f6f6ed0a4f8424fa13a72f57e90e369 WatchSource:0}: Error finding container b56572ac2aff94ee533053895a803de20f6f6ed0a4f8424fa13a72f57e90e369: Status 404 returned error can't find the container with id b56572ac2aff94ee533053895a803de20f6f6ed0a4f8424fa13a72f57e90e369 Oct 30 20:05:28.173608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:28.173565 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" event={"ID":"6262a9d5-36ff-4999-b1d6-1191b8303485","Type":"ContainerStarted","Data":"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b"} Oct 30 20:05:28.173608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:28.173618 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" event={"ID":"6262a9d5-36ff-4999-b1d6-1191b8303485","Type":"ContainerStarted","Data":"b56572ac2aff94ee533053895a803de20f6f6ed0a4f8424fa13a72f57e90e369"} Oct 30 20:05:28.295139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:28.295077 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podStartSLOduration=2.2950586 podStartE2EDuration="2.2950586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:28.29095016 +0000 UTC m=+1423.134621795" watchObservedRunningTime="2026-10-30 20:05:28.2950586 +0000 UTC m=+1423.138730229" Oct 30 20:05:28.937289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:28.937249 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.945672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:28.945646 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:33.885517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:33.885480 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.119206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:34.119172 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.946006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:36.945973 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:36.958612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:36.958590 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:37.238209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:37.238126 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:05:38.821714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:38.821620 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.094229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:40.094191 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.342964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:42.342931 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.784657 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:05:42.784607 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:05:43.856899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:43.856849 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.753509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:48.753460 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.522054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.522019 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:05:53.527549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.527528 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530394 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530372 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530420 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530625 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7nw\" (UniqueName: \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530664 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530789 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.530888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.530874 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.631725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631664 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7nw\" (UniqueName: \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.631930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631739 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.631930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.631930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631866 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.631930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631917 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.632138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.631979 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.632138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.632116 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.632204 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.632141 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.632279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.632257 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.633340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.633317 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"distinct-2-w6-f75272-kserve-self-signed-certs\"" Oct 30 20:05:53.633473 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.633367 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"default-dockercfg-qdntr\"" Oct 30 20:05:53.633560 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.633546 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"openshift-service-ca.crt\"" Oct 30 20:05:53.633640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.633622 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"kube-root-ca.crt\"" Oct 30 20:05:53.634200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.634177 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.636784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.636758 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:05:53.644453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.644433 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.721081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.721027 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.722367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.722339 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7nw\" (UniqueName: \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\") pod \"distinct-2-w6-f75272-kserve-5c8759578f-pr6r2\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:53.839940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:53.839850 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:54.316594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.316572 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:05:54.318630 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:05:54.318599 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7589ded7_4ca6_4489_803b_2dd4ec79bc12.slice/crio-dc16026e1e20386cb6f8cb3c05c5a2af41bb4e50a0a435dd9bc6db8506302353 WatchSource:0}: Error finding container dc16026e1e20386cb6f8cb3c05c5a2af41bb4e50a0a435dd9bc6db8506302353: Status 404 returned error can't find the container with id dc16026e1e20386cb6f8cb3c05c5a2af41bb4e50a0a435dd9bc6db8506302353 Oct 30 20:05:54.450773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.450733 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:54.561831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.561797 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" event={"ID":"7589ded7-4ca6-4489-803b-2dd4ec79bc12","Type":"ContainerStarted","Data":"db1dcaeb44ecc000e6ec7baeec5d6c189d17ca86cccdb5045240732ab4cfa5cb"} Oct 30 20:05:54.562270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.561838 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" event={"ID":"7589ded7-4ca6-4489-803b-2dd4ec79bc12","Type":"ContainerStarted","Data":"dc16026e1e20386cb6f8cb3c05c5a2af41bb4e50a0a435dd9bc6db8506302353"} Oct 30 20:05:54.682845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.682784 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podStartSLOduration=1.682764031 podStartE2EDuration="1.682764031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:54.678232073 +0000 UTC m=+1449.521903684" watchObservedRunningTime="2026-10-30 20:05:54.682764031 +0000 UTC m=+1449.526435660" Oct 30 20:05:54.851164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:54.847785 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:05:58.103590 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:58.103539 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:59.444464 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:05:59.444390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:03.840525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:03.840488 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:06:03.853584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:03.853560 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:06:04.439218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:04.439180 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.632365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:04.632339 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:06:09.465377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:09.465327 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.786062 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:06:12.786013 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:06:17.208804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.208740 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:17.209747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.209714 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:06:17.214091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.214066 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.275903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.275870 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:06:17.287036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287008 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs59t\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287046 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287149 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287170 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287197 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287236 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287433 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287491 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.287531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.287524 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.389613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389566 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs59t\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.389799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389623 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.389799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389715 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.389799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389744 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.389799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389784 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389841 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389938 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.389989 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390020 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390241 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390362 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390493 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.390938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390910 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.391043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.390940 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.392662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.392579 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.392894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.392874 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.504768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.487572 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.505758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.505725 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs59t\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:17.580977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:17.576963 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:18.385140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.381392 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:06:18.415914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.415853 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:18.416189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.416165 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:18.416367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.416339 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:18.845825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.845782 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" event={"ID":"5c45629c-ab72-45b4-b012-1a706baecf1c","Type":"ContainerStarted","Data":"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a"} Oct 30 20:06:18.852893 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.852862 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" event={"ID":"5c45629c-ab72-45b4-b012-1a706baecf1c","Type":"ContainerStarted","Data":"a8949a9f929a984d382ff6c614c5c2a94a13fed929e4ab34a2c882a14b4c5c16"} Oct 30 20:06:18.998963 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:18.998843 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podStartSLOduration=1.998820803 podStartE2EDuration="1.998820803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:18.998312577 +0000 UTC m=+1473.841984208" watchObservedRunningTime="2026-10-30 20:06:18.998820803 +0000 UTC m=+1473.842492435" Oct 30 20:06:19.587936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:19.587480 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:19.587936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:19.587607 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.587936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:19.587652 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:20.399725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:20.385823 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.618723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:20.610213 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.618723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:20.610275 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:20.618723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:20.615382 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:20.883908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:20.883806 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.582246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:21.582201 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.582459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:21.582276 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:22.581583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:22.581534 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.582092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:22.581618 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:23.582207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:23.582167 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:23.582745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:23.582216 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:24.582292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:24.582247 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:24.582842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:24.582311 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:25.581899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:25.581861 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.582074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:25.581924 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:25.872129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:25.868643 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.581780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:26.581675 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.581780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:26.581755 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 20:06:27.586326 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:27.586293 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:27.586772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:27.586685 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:27.587380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:27.587359 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:06:30.748616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:30.748551 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.578472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.578431 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:06:32.583779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.583758 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616400 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616451 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616535 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616581 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616655 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616683 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616801 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616850 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616839 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.616968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.616955 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5nl\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.627878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.627848 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:06:32.631720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.631690 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.677712 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.677664 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:06:32.718047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718014 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rrt\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718062 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5nl\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718140 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718168 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718212 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718229 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718263 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718304 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718419 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718469 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718510 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718531 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718576 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718577 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718591 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.718660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.718611 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.719044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719009 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.719109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719051 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.719109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719075 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.719212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719119 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.719339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719311 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.719455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719408 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.719770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.719750 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.721104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.721081 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.721605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.721579 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.735853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.735826 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:06:32.820351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820320 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rrt\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820530 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820480 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820600 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820530 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820600 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820589 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820669 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820822 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820854 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.820961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820919 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.820976 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.821001 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.821018 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821166 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.821061 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.821289 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.821653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.821630 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.823176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.823156 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.823331 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.823314 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.828995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.828945 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.829304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.829283 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5nl\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.882352 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.882318 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rrt\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.883016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.882993 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:32.896987 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.896965 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:32.945500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:32.945277 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:33.335791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.335757 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:06:33.373134 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:06:33.373105 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914be0c0_72ee_4b2e_9d47_4abaf5fadef4.slice/crio-16af2bf60f9a05cf7ef5a7ea522b91da79f2d47db34165e7985a75e64fc3f4bc WatchSource:0}: Error finding container 16af2bf60f9a05cf7ef5a7ea522b91da79f2d47db34165e7985a75e64fc3f4bc: Status 404 returned error can't find the container with id 16af2bf60f9a05cf7ef5a7ea522b91da79f2d47db34165e7985a75e64fc3f4bc Oct 30 20:06:33.375446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.375414 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:33.375536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.375480 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:33.375536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.375527 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:33.395897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.395821 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:06:33.410931 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:06:33.410904 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193a8f69_caf0_4429_9d15_f7f3077e5f07.slice/crio-55d783008da8b9d9e161f67b7f581c6657c9256cb2259e7a12398f9563ea305d WatchSource:0}: Error finding container 55d783008da8b9d9e161f67b7f581c6657c9256cb2259e7a12398f9563ea305d: Status 404 returned error can't find the container with id 55d783008da8b9d9e161f67b7f581c6657c9256cb2259e7a12398f9563ea305d Oct 30 20:06:33.413420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.413385 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:33.413508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.413459 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:33.413576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:33.413514 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:34.105682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.104102 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" event={"ID":"193a8f69-caf0-4429-9d15-f7f3077e5f07","Type":"ContainerStarted","Data":"8b38168b8209e515cd79886e0daa1cbfd0c439137145631f3e4f0a3f139cc711"} Oct 30 20:06:34.105682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.104209 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" event={"ID":"193a8f69-caf0-4429-9d15-f7f3077e5f07","Type":"ContainerStarted","Data":"55d783008da8b9d9e161f67b7f581c6657c9256cb2259e7a12398f9563ea305d"} Oct 30 20:06:34.106718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.106465 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" event={"ID":"914be0c0-72ee-4b2e-9d47-4abaf5fadef4","Type":"ContainerStarted","Data":"8568cd761d0d9f0c9f433b4fddde9ff6edfee715e4581052374c2a6d4cffa9ba"} Oct 30 20:06:34.106718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.106519 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" event={"ID":"914be0c0-72ee-4b2e-9d47-4abaf5fadef4","Type":"ContainerStarted","Data":"16af2bf60f9a05cf7ef5a7ea522b91da79f2d47db34165e7985a75e64fc3f4bc"} Oct 30 20:06:34.304724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.303919 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podStartSLOduration=2.303899429 podStartE2EDuration="2.303899429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:34.3029624 +0000 UTC m=+1489.146634040" watchObservedRunningTime="2026-10-30 20:06:34.303899429 +0000 UTC m=+1489.147571060" Oct 30 20:06:34.900791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.900751 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:34.900999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.900944 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:34.901074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.901012 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:34.951725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.951317 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:34.951725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.951430 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:34.951725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:34.951486 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:35.900726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:35.900106 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:35.900726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:35.900183 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:35.949724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:35.947048 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:35.949724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:35.947115 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:36.134725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:36.131915 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.903755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:36.900078 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:36.903755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:36.900143 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:36.945586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:36.945554 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:36.945792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:36.945603 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:37.898107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:37.898070 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:37.898392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:37.898362 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:37.949830 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:37.949793 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:37.950398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:37.950363 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:38.899767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:38.898181 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:38.899767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:38.898241 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:38.953728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:38.951528 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:38.953728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:38.951587 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:39.898371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:39.898329 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:06:39.898566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:39.898403 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:06:39.946221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:39.946183 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:06:39.946381 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:39.946260 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:06:40.901787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.901666 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.933722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.931125 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:40.933722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.932014 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:40.936850 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.936795 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:06:40.951891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.951860 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:40.952293 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.952261 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:40.953744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:40.953694 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:06:41.186772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:41.177607 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podStartSLOduration=9.177587188 podStartE2EDuration="9.177587188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:34.46504876 +0000 UTC m=+1489.308720390" watchObservedRunningTime="2026-10-30 20:06:41.177587188 +0000 UTC m=+1496.021258818" Oct 30 20:06:41.193778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:41.193746 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.696678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:42.696644 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.786411 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:06:42.786372 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:06:46.252510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.252467 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:46.424606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.424572 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:06:46.428756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.428736 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524245 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524296 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524389 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524434 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524524 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f5k\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524546 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.524573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524572 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.525030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524714 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.525030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.524745 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626362 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626402 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626467 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626504 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626544 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626572 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626637 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f5k\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626671 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.626889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626745 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.627050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.626962 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.627108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.627094 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.627158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.627123 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.627266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.627238 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.627372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.627271 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.629119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.629094 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.629430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.629407 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.654817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.654788 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:06:46.696743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.692637 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f5k\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.696743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.695426 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:46.743681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:46.743652 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:47.203592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.203556 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:06:47.212753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.212682 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:47.212871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.212785 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:47.212871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.212822 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:06:47.302367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.302333 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" event={"ID":"fa4bccdc-1419-4643-bcef-55dc576f1bad","Type":"ContainerStarted","Data":"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef"} Oct 30 20:06:47.302845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.302393 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" event={"ID":"fa4bccdc-1419-4643-bcef-55dc576f1bad","Type":"ContainerStarted","Data":"23a9cdc812282ada1cb3c24006829ebedb8d8d7db1a0035f14dcda0ec4730f3e"} Oct 30 20:06:47.421723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.418527 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podStartSLOduration=1.418506973 podStartE2EDuration="1.418506973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:47.416933639 +0000 UTC m=+1502.260605266" watchObservedRunningTime="2026-10-30 20:06:47.418506973 +0000 UTC m=+1502.262178603" Oct 30 20:06:47.744632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:47.744595 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:48.753721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:48.752884 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:48.753721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:48.752951 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:49.746925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:49.746888 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:49.747215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:49.747184 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:50.751273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:50.751232 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:50.752593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:50.752557 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:50.886929 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:50.886830 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.364733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:51.354118 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.763722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:51.755991 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:06:51.806732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:51.804625 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:51.806732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:51.804691 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:52.081873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.081766 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:52.097488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.097450 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:06:52.227881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.227840 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:06:52.356089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.355996 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:52.357573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.357547 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:06:52.768722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.733084 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.808763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.770159 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:52.808763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.783852 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:52.808763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.783912 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:52.808763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:52.784392 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:06:53.120784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.116823 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:53.246819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.246778 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:53.260724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.247375 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:06:53.302172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.302141 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:06:53.323284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.322308 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:06:53.326416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.326393 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:06:53.328276 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.328255 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:06:53.744646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.744615 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:53.744826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:53.744667 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:54.744799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:54.744761 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:54.745189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:54.744816 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:55.744369 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:55.744327 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:06:55.744571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:55.744381 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:06:57.751513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:57.751474 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:57.752103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:57.752070 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:57.767372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:57.767336 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:57.768199 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:57.768172 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:57.772358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:57.772339 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:06:58.110603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:06:58.110522 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:01.539408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.539378 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:07:01.544474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.544453 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594318 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594420 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594543 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594515 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594598 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594729 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z2n\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594773 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.594910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594886 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.595009 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.594981 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.595057 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.595045 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.654154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.654114 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:07:01.696529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696498 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696546 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z2n\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696575 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696620 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696721 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696748 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.696862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696826 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696874 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.696908 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.697263 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.697540 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.697731 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.697744 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.697875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.697854 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.699379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.699357 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.699539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.699521 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.758048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.758023 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.758290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.758274 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z2n\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-knxfm\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:01.858907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:01.858830 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:02.295847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.295822 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:07:02.297474 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:07:02.297442 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95da0860_06ba_431f_87d7_d77554fd280d.slice/crio-a693af19969cdd779cbb7ef85c17a514a24e8dda76d61e2d42e62065f9500ad7 WatchSource:0}: Error finding container a693af19969cdd779cbb7ef85c17a514a24e8dda76d61e2d42e62065f9500ad7: Status 404 returned error can't find the container with id a693af19969cdd779cbb7ef85c17a514a24e8dda76d61e2d42e62065f9500ad7 Oct 30 20:07:02.299854 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.299821 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:02.299947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.299887 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:02.299947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.299919 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:02.739454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.739419 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.986875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:02.986836 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.135901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.126885 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:07:03.158765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.141443 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.226534 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6tc\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.226771 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.226846 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.227338 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.227758 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.228107 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.228172 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.228201 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.232760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.228465 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.257721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.235515 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.257721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.239030 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:07:03.320621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.320589 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" event={"ID":"95da0860-06ba-431f-87d7-d77554fd280d","Type":"ContainerStarted","Data":"2ff8654519e5d2c1bf630db1dbd290e27fa7fd49c1b579c40845b7aa692b2256"} Oct 30 20:07:03.320824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.320646 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" event={"ID":"95da0860-06ba-431f-87d7-d77554fd280d","Type":"ContainerStarted","Data":"a693af19969cdd779cbb7ef85c17a514a24e8dda76d61e2d42e62065f9500ad7"} Oct 30 20:07:03.329845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.329815 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.329979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.329886 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330034 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330180 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330260 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330244 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330265 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330273 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330327 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330340 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.330580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.330559 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.331075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.331050 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.331411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.331385 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6tc\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.331503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.331424 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.331731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.331694 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.334362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.334339 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.335263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.335241 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.393413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.393379 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6tc\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.393564 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.393519 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.472180 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.472108 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podStartSLOduration=2.472087481 podStartE2EDuration="2.472087481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:03.472032277 +0000 UTC m=+1518.315703908" watchObservedRunningTime="2026-10-30 20:07:03.472087481 +0000 UTC m=+1518.315759111" Oct 30 20:07:03.620849 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.620819 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:03.871723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.871481 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:03.874773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.874102 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:03.874773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:03.874187 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:04.083181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.081974 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:07:04.087790 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:07:04.086552 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c844a2_dfd9_45bf_84c5_0b27690a0eb9.slice/crio-80a6894e64d2fa02a879583861a1450d8ba40905b72043582f6120eee5767bb4 WatchSource:0}: Error finding container 80a6894e64d2fa02a879583861a1450d8ba40905b72043582f6120eee5767bb4: Status 404 returned error can't find the container with id 80a6894e64d2fa02a879583861a1450d8ba40905b72043582f6120eee5767bb4 Oct 30 20:07:04.089947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.089906 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:04.090055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.089993 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:04.090055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.090036 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:04.329082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.329038 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" event={"ID":"71c844a2-dfd9-45bf-84c5-0b27690a0eb9","Type":"ContainerStarted","Data":"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2"} Oct 30 20:07:04.329248 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.329110 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" event={"ID":"71c844a2-dfd9-45bf-84c5-0b27690a0eb9","Type":"ContainerStarted","Data":"80a6894e64d2fa02a879583861a1450d8ba40905b72043582f6120eee5767bb4"} Oct 30 20:07:04.621931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.621898 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:04.863766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.863153 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:04.863766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:04.863224 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:05.623131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:05.623102 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:05.623619 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:05.623592 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:05.868837 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:05.868797 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:05.869172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:05.869138 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:06.621800 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:06.621766 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:06.622001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:06.621841 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:06.863725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:06.860097 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:06.863725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:06.860161 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:07.621776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:07.621746 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:07.622029 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:07.622009 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:07.862175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:07.862129 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:07.862452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:07.862424 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:08.588721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:08.581418 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:08.624731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:08.622197 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:08.624731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:08.622255 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:08.860780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:08.860073 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:08.860780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:08.860133 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:07:09.621212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:09.621176 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:09.621607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:09.621238 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:10.024532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.024495 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:10.025182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.025031 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:10.025855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.025831 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:07:10.201540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.201484 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podStartSLOduration=7.201468938 podStartE2EDuration="7.201468938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:04.466269963 +0000 UTC m=+1519.309941593" watchObservedRunningTime="2026-10-30 20:07:10.201468938 +0000 UTC m=+1525.045140576" Oct 30 20:07:10.622101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.622048 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:07:10.622544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:10.622153 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:07:11.799745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:11.799687 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:11.800339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:11.800195 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:11.800991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:11.800970 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:07:12.786508 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:07:12.786476 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:07:13.643739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:13.643694 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:17.025401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:17.025368 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:18.254527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.254491 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:07:18.259721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.259685 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.366606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.366567 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:07:18.426683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.426647 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.426865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.426810 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.426921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.426902 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.426979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.426963 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.427092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.427068 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr95z\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.427233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.427213 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.427300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.427286 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.427348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.427312 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.427398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.427377 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528545 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528617 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528670 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528737 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528767 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr95z\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.528874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528841 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528930 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528959 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.528984 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.529090 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.529113 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.529325 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.529350 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.529901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.529744 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.531279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.531254 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.531485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.531469 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.589631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.589602 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.589943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.589925 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr95z\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:18.691287 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.691210 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:18.873946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:18.873866 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:19.312186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.312153 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:07:19.312599 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:07:19.312496 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a5eb43_b935_4ffe_8044_388cb9267850.slice/crio-85bcb6f5bb259a96cd86fdeea6cf769448981dad8c5aa02ba6c7620a5fb33976 WatchSource:0}: Error finding container 85bcb6f5bb259a96cd86fdeea6cf769448981dad8c5aa02ba6c7620a5fb33976: Status 404 returned error can't find the container with id 85bcb6f5bb259a96cd86fdeea6cf769448981dad8c5aa02ba6c7620a5fb33976 Oct 30 20:07:19.315534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.315502 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.315615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.315566 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.315615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.315597 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:19.492245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.491673 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" event={"ID":"74a5eb43-b935-4ffe-8044-388cb9267850","Type":"ContainerStarted","Data":"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b"} Oct 30 20:07:19.492245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.491750 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" event={"ID":"74a5eb43-b935-4ffe-8044-388cb9267850","Type":"ContainerStarted","Data":"85bcb6f5bb259a96cd86fdeea6cf769448981dad8c5aa02ba6c7620a5fb33976"} Oct 30 20:07:19.620852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.620764 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podStartSLOduration=1.620749751 podStartE2EDuration="1.620749751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:19.618208305 +0000 UTC m=+1534.461879935" watchObservedRunningTime="2026-10-30 20:07:19.620749751 +0000 UTC m=+1534.464421390" Oct 30 20:07:19.875301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:19.875215 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:20.912722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:20.896827 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.912722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:20.896899 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:21.896722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:21.879556 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:21.896722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:21.879641 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:22.162779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:22.162343 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:22.345725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:22.335694 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:22.356889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:22.356843 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:22.882728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:22.882153 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:22.882728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:22.882214 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:23.183728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.183233 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.184250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.184103 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:07:23.343732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.343318 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.353509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.352737 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.379723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.379138 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.752738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.752089 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:07:23.752738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.752159 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:07:23.915870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.915830 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:23.916156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.916125 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.975724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.955967 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:23.975724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:23.956023 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:24.188168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.188127 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.344828 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.344779 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.391725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.372238 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.391725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.385783 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.486778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.485806 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:07:24.532731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.529033 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.649618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.649545 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:24.859781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.848802 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:24.947723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.919861 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:24.947995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.947959 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:25.194000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:24.962512 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.194773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.012310 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:25.218052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.194995 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:25.274759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.232899 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:25.372829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.368006 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:25.408751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.386239 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:07:25.408751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:25.387011 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:26.208735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.206883 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:26.208735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.207140 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:26.407822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.345083 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:26.407822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.368110 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.500923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.500811 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:07:26.533606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.503510 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:07:26.724725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.698570 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:07:26.755267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.755169 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-x4wvm" podUID="36dc7fbd-ec30-4beb-8b63-6c82b7948da4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 30 20:07:26.797989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.777547 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:26.812848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.812803 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:26.816730 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:07:26.784977 4524 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.22s" Oct 30 20:07:26.907766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.882969 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:26.908069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:26.908033 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:27.040387 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.040307 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:07:27.042846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.042823 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:07:27.104720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.104449 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:07:27.110146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.110120 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:07:27.877078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.877039 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:27.877509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:27.877105 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:28.879174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:28.875027 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:28.879174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:28.875089 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:29.874760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:29.874691 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:07:29.874930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:29.874792 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:07:31.105344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:31.105311 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:31.105824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:31.105810 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:31.106801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:31.106782 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:07:31.504329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:31.504289 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:31.964369 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:31.964326 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:32.524829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:32.524776 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:36.698388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:36.698348 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:41.618299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:41.618256 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.554671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:42.554628 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.838085 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:07:42.838001 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:07:43.091341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:43.091263 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:46.775999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:46.775958 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:47.196608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.196574 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:07:47.201746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.201716 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.247407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.247382 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:07:47.251109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.251093 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.268799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.268776 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269041 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269170 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269088 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269170 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269117 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269235 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269285 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269334 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269361 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269389 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92wx\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269436 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269488 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnvq\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269513 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269558 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269641 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269667 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269729 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.269906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269775 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.269906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.269823 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.304968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.304936 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:07:47.350604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.350572 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:07:47.371028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.370990 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnvq\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371038 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371082 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371162 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371189 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371215 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371257 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371301 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371418 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371478 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371519 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371544 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371653 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371694 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371814 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.371924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371838 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92wx\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.371924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.371891 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.372144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.372114 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.372263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.372239 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.372363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.372341 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.372662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.372640 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.372980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.372957 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.373250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.373227 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.373334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.373279 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.373547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.373524 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.373815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.373792 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.374014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.373995 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.375558 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.375524 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.376585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.376563 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.377045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.377021 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.379022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.378997 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.438692 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.438664 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnvq\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.439000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.438980 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.457583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.457521 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92wx\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.459178 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.459150 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.518067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.518042 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:47.568747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.568488 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:47.969391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.969357 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:07:47.993727 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:07:47.993682 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71fc396_ea17_426f_868d_05d5fafce8ae.slice/crio-e566c1cbca99df2b16f0c30e0afe0262a4717c19eccd5634af7d2e8597650e06 WatchSource:0}: Error finding container e566c1cbca99df2b16f0c30e0afe0262a4717c19eccd5634af7d2e8597650e06: Status 404 returned error can't find the container with id e566c1cbca99df2b16f0c30e0afe0262a4717c19eccd5634af7d2e8597650e06 Oct 30 20:07:47.996100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.996067 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:47.996209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.996135 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:47.996209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:47.996173 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:48.011691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.011666 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:07:48.012413 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:07:48.012386 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcf7254_2b05_412d_b788_5a48a48205da.slice/crio-bf36fb908589563cafbe5b1d54942054ba934b4d761405ccdf5a6e7f20ea15e5 WatchSource:0}: Error finding container bf36fb908589563cafbe5b1d54942054ba934b4d761405ccdf5a6e7f20ea15e5: Status 404 returned error can't find the container with id bf36fb908589563cafbe5b1d54942054ba934b4d761405ccdf5a6e7f20ea15e5 Oct 30 20:07:48.014844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.014817 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:48.014921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.014879 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:48.014921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.014910 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:48.938963 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.938924 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" event={"ID":"e71fc396-ea17-426f-868d-05d5fafce8ae","Type":"ContainerStarted","Data":"9267fe4502441f6ce5e721d306b21e8cbdd4c787f9c1e47718efde87aef860a2"} Oct 30 20:07:48.939155 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.938983 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" event={"ID":"e71fc396-ea17-426f-868d-05d5fafce8ae","Type":"ContainerStarted","Data":"e566c1cbca99df2b16f0c30e0afe0262a4717c19eccd5634af7d2e8597650e06"} Oct 30 20:07:48.940581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.940552 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" event={"ID":"ddcf7254-2b05-412d-b788-5a48a48205da","Type":"ContainerStarted","Data":"56803420a2845ca4a772f3d3c1334e9780c56f1e3e86506e56c85bb0e1b105c8"} Oct 30 20:07:48.940759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:48.940593 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" event={"ID":"ddcf7254-2b05-412d-b788-5a48a48205da","Type":"ContainerStarted","Data":"bf36fb908589563cafbe5b1d54942054ba934b4d761405ccdf5a6e7f20ea15e5"} Oct 30 20:07:49.170723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.162460 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podStartSLOduration=2.16244039 podStartE2EDuration="2.16244039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:49.11099214 +0000 UTC m=+1563.954663782" watchObservedRunningTime="2026-10-30 20:07:49.16244039 +0000 UTC m=+1564.006112020" Oct 30 20:07:49.260780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.258322 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podStartSLOduration=2.258298499 podStartE2EDuration="2.258298499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:49.245212145 +0000 UTC m=+1564.088883775" watchObservedRunningTime="2026-10-30 20:07:49.258298499 +0000 UTC m=+1564.101970129" Oct 30 20:07:49.518680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.518593 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:49.519994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.519968 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:49.520065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.520031 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:49.569692 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.569324 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:49.571351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.571318 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:49.571605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:49.571562 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:50.520073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:50.520043 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:50.520469 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:50.520133 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:50.586747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:50.586451 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:50.586747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:50.586508 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:51.519112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:51.519082 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:51.519334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:51.519315 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:51.592768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:51.587266 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:51.592768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:51.587324 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:52.123334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:52.123253 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:52.519083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:52.519041 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:52.519281 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:52.519097 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:52.573723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:52.573195 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:52.573723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:52.573255 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:53.531721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:53.529567 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:53.531721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:53.529617 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:53.574124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:53.574087 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:53.574409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:53.574378 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:54.519392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:54.519360 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:07:54.519606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:54.519412 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:07:54.570057 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:54.570023 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:07:54.570542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:54.570078 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:07:56.069761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.065025 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:56.069761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.066613 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:56.092173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.092145 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:07:56.092578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.092556 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:56.093664 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.093645 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:56.097058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:56.096055 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:07:57.455726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:57.453283 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:57.455726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:57.454198 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:57.551737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:57.517134 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:57.953723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:57.950846 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.261780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.248780 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.405724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.403813 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.456823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.456779 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.462721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.457778 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:07:58.529783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.518896 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.529783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.520422 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:07:58.646884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.646837 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:58.647165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.647130 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.927726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:58.924278 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:59.228776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.224820 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:59.323003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.253017 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.323003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.252822 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.323003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.253652 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:07:59.418659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.414153 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.478721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.478496 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.540441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.538520 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.623493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:07:59.612337 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:08:00.287724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.265595 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.432722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.431838 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.494955 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.494203 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.543443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.541821 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.553346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.553317 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:08:00.646724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:00.606835 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:08:01.072743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:01.067263 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:08:01.559726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:01.544105 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:01.762174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:01.762061 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:08:05.151145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:05.150233 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:07.576240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.576207 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 20:08:07.576718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.576400 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" containerID="cri-o://818c95e9c1a6b040872248460866df4ed8c89dca21badea782371898f576d73b" gracePeriod=30 Oct 30 20:08:07.626682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.626650 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:08:07.626910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.626885 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" containerID="cri-o://efd7606689ebb05f266c8fd5ce93a5ad1ee5588eca3a0dca9c278f6ece50790b" gracePeriod=30 Oct 30 20:08:07.630341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.630314 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:08:07.630550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.630526 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" containerID="cri-o://37f6a7665d372fd749a5a528dc5fb88e7f94c1b2e56bbaf467b8540181cab9c4" gracePeriod=30 Oct 30 20:08:07.675750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.675720 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:08:07.675900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:07.675882 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" containerID="cri-o://c035571b1d30b92d5a18aff75abaf74fdbb1f4d6c26a365954fcf03e8b687bb8" gracePeriod=30 Oct 30 20:08:10.146726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:10.145648 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:11.296722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:11.280275 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:12.794053 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:08:12.794009 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:08:12.939091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.939060 4524 generic.go:356] "Generic (PLEG): container finished" podID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerID="818c95e9c1a6b040872248460866df4ed8c89dca21badea782371898f576d73b" exitCode=0 Oct 30 20:08:12.939231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.939149 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" event={"ID":"3517d6d9-a773-4377-bff2-8ef3a0b24893","Type":"ContainerDied","Data":"818c95e9c1a6b040872248460866df4ed8c89dca21badea782371898f576d73b"} Oct 30 20:08:12.939231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.939201 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" event={"ID":"3517d6d9-a773-4377-bff2-8ef3a0b24893","Type":"ContainerDied","Data":"7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592"} Oct 30 20:08:12.939231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.939223 4524 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be2c033f5f22e0b726cbffe2d752834c25084f2caed66df98b4c6f2f337e592" Oct 30 20:08:12.940710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.940680 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 20:08:12.940947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.940922 4524 generic.go:356] "Generic (PLEG): container finished" podID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerID="37f6a7665d372fd749a5a528dc5fb88e7f94c1b2e56bbaf467b8540181cab9c4" exitCode=0 Oct 30 20:08:12.941071 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.941051 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" event={"ID":"8c515898-62e1-4467-9de1-b2aa3a1e627b","Type":"ContainerDied","Data":"37f6a7665d372fd749a5a528dc5fb88e7f94c1b2e56bbaf467b8540181cab9c4"} Oct 30 20:08:12.943081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.943050 4524 generic.go:356] "Generic (PLEG): container finished" podID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerID="efd7606689ebb05f266c8fd5ce93a5ad1ee5588eca3a0dca9c278f6ece50790b" exitCode=0 Oct 30 20:08:12.943236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.943155 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" event={"ID":"59604139-4a1e-49a7-bdeb-53825ca1fb7a","Type":"ContainerDied","Data":"efd7606689ebb05f266c8fd5ce93a5ad1ee5588eca3a0dca9c278f6ece50790b"} Oct 30 20:08:12.945383 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.945359 4524 generic.go:356] "Generic (PLEG): container finished" podID="1924a830-7a78-47f0-a825-37e523111f58" containerID="c035571b1d30b92d5a18aff75abaf74fdbb1f4d6c26a365954fcf03e8b687bb8" exitCode=0 Oct 30 20:08:12.945492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.945391 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" event={"ID":"1924a830-7a78-47f0-a825-37e523111f58","Type":"ContainerDied","Data":"c035571b1d30b92d5a18aff75abaf74fdbb1f4d6c26a365954fcf03e8b687bb8"} Oct 30 20:08:12.985246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985203 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985274 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985420 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985483 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985512 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985551 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985604 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.985840 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985808 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.986006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.985846 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\") pod \"3517d6d9-a773-4377-bff2-8ef3a0b24893\" (UID: \"3517d6d9-a773-4377-bff2-8ef3a0b24893\") " Oct 30 20:08:12.986875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.986805 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.987367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.987038 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.987367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.987052 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.987367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.987328 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:12.987612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.987519 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.990603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.989614 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:12.990603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.989841 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:12.990603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.990443 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "kube-api-access-h52zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:12.991666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.991643 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token" pod "3517d6d9-a773-4377-bff2-8ef3a0b24893" (UID: "3517d6d9-a773-4377-bff2-8ef3a0b24893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:12.999822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:12.999436 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:08:13.063287 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.063264 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:08:13.067626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.067607 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:08:13.087865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.087834 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.087990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.087884 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.087990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.087930 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.087997 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088047 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088078 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088137 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.088264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088207 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088254 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088272 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.088430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088308 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\") pod \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\" (UID: \"59604139-4a1e-49a7-bdeb-53825ca1fb7a\") " Oct 30 20:08:13.088430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088323 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.088578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088489 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.088748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.088724 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:13.089061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089039 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089069 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089085 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089100 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089113 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089127 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089142 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089159 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089173 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089187 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3517d6d9-a773-4377-bff2-8ef3a0b24893-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089201 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089215 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3517d6d9-a773-4377-bff2-8ef3a0b24893-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089230 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.089425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.089245 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52zf\" (UniqueName: \"kubernetes.io/projected/3517d6d9-a773-4377-bff2-8ef3a0b24893-kube-api-access-h52zf\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.090312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.090288 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.090391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.090307 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:13.090391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.090365 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "kube-api-access-phdpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.090499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.090486 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token" pod "59604139-4a1e-49a7-bdeb-53825ca1fb7a" (UID: "59604139-4a1e-49a7-bdeb-53825ca1fb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.190675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190642 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.190866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190720 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.190866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190834 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.190992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190890 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.190992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190909 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.190992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190933 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.190992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190957 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.190992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.190981 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191048 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191155 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191164 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191203 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191220 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191232 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191261 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191341 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191406 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191420 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191430 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191466 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191562 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191570 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.191634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191612 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.192020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191642 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\") pod \"1924a830-7a78-47f0-a825-37e523111f58\" (UID: \"1924a830-7a78-47f0-a825-37e523111f58\") " Oct 30 20:08:13.192020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.191768 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\") pod \"8c515898-62e1-4467-9de1-b2aa3a1e627b\" (UID: \"8c515898-62e1-4467-9de1-b2aa3a1e627b\") " Oct 30 20:08:13.192119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192046 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:13.192273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192250 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.192342 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192274 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.192614 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192582 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192614 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192610 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdpv\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-kube-api-access-phdpv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192626 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192638 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192651 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1924a830-7a78-47f0-a825-37e523111f58-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192665 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192680 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192693 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192720 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192733 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59604139-4a1e-49a7-bdeb-53825ca1fb7a-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192746 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c515898-62e1-4467-9de1-b2aa3a1e627b-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192759 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192774 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.192815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.192788 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.193931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.193904 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:13.194047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.193960 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.194047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.193966 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:13.194047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.194014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "kube-api-access-zqcvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.194268 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.194242 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "kube-api-access-qfcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.194354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.194280 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token" pod "8c515898-62e1-4467-9de1-b2aa3a1e627b" (UID: "8c515898-62e1-4467-9de1-b2aa3a1e627b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.194970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.194948 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:13.195427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.195412 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token" pod "1924a830-7a78-47f0-a825-37e523111f58" (UID: "1924a830-7a78-47f0-a825-37e523111f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301273 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1924a830-7a78-47f0-a825-37e523111f58-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301311 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301326 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcnk\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-kube-api-access-qfcnk\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301338 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301351 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcvh\" (UniqueName: \"kubernetes.io/projected/8c515898-62e1-4467-9de1-b2aa3a1e627b-kube-api-access-zqcvh\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301363 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c515898-62e1-4467-9de1-b2aa3a1e627b-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301376 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1924a830-7a78-47f0-a825-37e523111f58-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.301419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.301389 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1924a830-7a78-47f0-a825-37e523111f58-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:08:13.953207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.953148 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" Oct 30 20:08:13.953678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.953173 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267" event={"ID":"8c515898-62e1-4467-9de1-b2aa3a1e627b","Type":"ContainerDied","Data":"77d2a0a51236b3113827933888392fbc9e935661a58ee34aa4325bed282ec2c6"} Oct 30 20:08:13.953678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.953323 4524 scope.go:122] "RemoveContainer" containerID="37f6a7665d372fd749a5a528dc5fb88e7f94c1b2e56bbaf467b8540181cab9c4" Oct 30 20:08:13.954745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.954724 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" Oct 30 20:08:13.954844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.954736 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs" event={"ID":"59604139-4a1e-49a7-bdeb-53825ca1fb7a","Type":"ContainerDied","Data":"a63c044a7cfd62e10f4898ad3a2785f051a5fe479f811a13904e4c7a8521f3a4"} Oct 30 20:08:13.956253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.956227 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" Oct 30 20:08:13.956360 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.956262 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82" event={"ID":"1924a830-7a78-47f0-a825-37e523111f58","Type":"ContainerDied","Data":"3ed0babbdd80255e5d98ab6cf0c57af715320adc2a2b2f73e8100b408857db6b"} Oct 30 20:08:13.956430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.956396 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj" Oct 30 20:08:13.969016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.968988 4524 scope.go:122] "RemoveContainer" containerID="efd7606689ebb05f266c8fd5ce93a5ad1ee5588eca3a0dca9c278f6ece50790b" Oct 30 20:08:13.977693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:13.977676 4524 scope.go:122] "RemoveContainer" containerID="c035571b1d30b92d5a18aff75abaf74fdbb1f4d6c26a365954fcf03e8b687bb8" Oct 30 20:08:14.124286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.124255 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 20:08:14.133266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.133237 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-745qj"] Oct 30 20:08:14.249734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.249647 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:08:14.253413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.253392 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tn267"] Oct 30 20:08:14.363483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.363447 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:08:14.372137 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.372111 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fpj82"] Oct 30 20:08:14.483446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.483401 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:08:14.490568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:14.490533 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8pkrs"] Oct 30 20:08:15.334953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:15.329474 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1924a830-7a78-47f0-a825-37e523111f58" path="/var/lib/kubelet/pods/1924a830-7a78-47f0-a825-37e523111f58/volumes" Oct 30 20:08:15.334953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:15.330328 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" path="/var/lib/kubelet/pods/3517d6d9-a773-4377-bff2-8ef3a0b24893/volumes" Oct 30 20:08:15.334953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:15.330951 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" path="/var/lib/kubelet/pods/59604139-4a1e-49a7-bdeb-53825ca1fb7a/volumes" Oct 30 20:08:15.334953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:15.332096 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" path="/var/lib/kubelet/pods/8c515898-62e1-4467-9de1-b2aa3a1e627b/volumes" Oct 30 20:08:15.347657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:15.347625 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:20.387729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:20.384322 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:25.311723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:25.304611 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:30.515725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:30.509823 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:08:30.564725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:30.561821 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:08:30.564725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:30.562732 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:31.069859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.069805 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:31.514726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.510793 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:31.514726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.511555 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:08:31.589728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.577379 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:31.589728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.578528 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:08:31.589728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.579098 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:08:31.589728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:31.579441 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:08:32.126725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.098341 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:08:32.126725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.100404 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:08:32.174643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.173265 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:32.174643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.173345 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:32.376051 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.376013 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:08:32.420754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.409412 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" Oct 30 20:08:32.568824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.568790 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" Oct 30 20:08:32.572799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.572767 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" Oct 30 20:08:32.580771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.580734 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" Oct 30 20:08:32.580994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.580976 4524 cpu_manager.go:414] "containerMap: removing container" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" Oct 30 20:08:32.581095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.581083 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" Oct 30 20:08:32.581187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.581177 4524 cpu_manager.go:414] "containerMap: removing container" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" Oct 30 20:08:32.581259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.581249 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" Oct 30 20:08:32.584722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.581605 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3517d6d9-a773-4377-bff2-8ef3a0b24893" containerName="istio-proxy" Oct 30 20:08:32.593812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.591226 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c515898-62e1-4467-9de1-b2aa3a1e627b" containerName="istio-proxy" Oct 30 20:08:32.593812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.591249 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="1924a830-7a78-47f0-a825-37e523111f58" containerName="istio-proxy" Oct 30 20:08:32.593812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.591270 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="59604139-4a1e-49a7-bdeb-53825ca1fb7a" containerName="istio-proxy" Oct 30 20:08:32.615945 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.612842 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:32.616220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.616187 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:32.659967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.449220 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:32.660289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.540780 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:32.685274 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.685071 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794068 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794148 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794268 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794319 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794368 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trkc\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794413 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794442 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794499 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.797724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.794529 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.845728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.799860 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:08:32.938901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.938806 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939144 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939353 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939503 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939630 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trkc\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939780 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.939927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.939911 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.940059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.940044 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.940174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.940158 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:32.984653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.983579 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.003450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.995019 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.003450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.995265 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.003450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.995486 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.003450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:32.995727 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.119725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.009003 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.143847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.142577 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:33.213777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.205861 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:08:33.215020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.214835 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.214937 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkfv\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.214974 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215004 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215140 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215203 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215234 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215296 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.215539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.215346 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.263807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.235911 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.317660 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.317782 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkfv\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.317819 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.317848 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.317980 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.318049 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.318082 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.318149 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.318203 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.318935 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.319218 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.319821 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.320067 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.336758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.336004 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.664724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.662137 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trkc\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.664724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.662968 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.665886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.665849 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:33.676914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.676876 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:33.681074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.681047 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:33.704762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.684502 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.743731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.732899 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.946773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.915659 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkfv\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:33.980537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.964187 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:08:33.982502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:33.982465 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:34.138725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.104943 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:34.386772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.370074 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:08:34.613297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.613262 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:34.737723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.730120 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:08:34.742118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.742078 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:08:34.763423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.762777 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:34.811779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.786016 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:08:34.905896 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.905846 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:34.966799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:34.966743 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:35.458722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:35.455002 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:08:35.761805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:35.760945 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:08:35.761805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:35.761367 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:08:36.053156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.013201 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:36.309305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.296239 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:36.466724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.458071 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:36.815758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.766761 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:36.815758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.767171 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:08:36.843725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:36.820802 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:08:37.088628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:37.088311 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:37.398252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:37.398204 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:37.500956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:37.458833 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:08:37.811690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:37.811644 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:38.042787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.030870 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:38.042787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.031041 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:08:38.317431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.317321 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:38.552784 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:08:38.552747 4524 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.176s" Oct 30 20:08:38.553173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.553151 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:08:38.558098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.558043 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.558503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.558452 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.558818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.558775 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.639996 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.639948 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:08:38.836729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.833584 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.836729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.834508 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.836729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.835174 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:38.894223 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:38.865022 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:39.035236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.034796 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:39.132053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.128999 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:08:39.149064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.148974 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" event={"ID":"dfbf3037-13bd-4455-b563-f2c43171e660","Type":"ContainerStarted","Data":"1783aff7dfaa9ab52a679e68e6c589afc36478aaf120f6d107d9c25065f79db4"} Oct 30 20:08:39.149461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.149429 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" event={"ID":"73c96e0c-4fc8-422f-8ca0-e366fa90e63e","Type":"ContainerStarted","Data":"f22eaf32685217aee2294ccce3fd9f7100523ba3dcb8332678a54a1c303c9b77"} Oct 30 20:08:39.404805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.394682 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:39.556547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.552041 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:08:39.587363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.587313 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:08:39.588668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:39.588648 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:08:40.145138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.145100 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" event={"ID":"dfbf3037-13bd-4455-b563-f2c43171e660","Type":"ContainerStarted","Data":"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9"} Oct 30 20:08:40.147850 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.147812 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" event={"ID":"73c96e0c-4fc8-422f-8ca0-e366fa90e63e","Type":"ContainerStarted","Data":"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428"} Oct 30 20:08:40.266879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.266815 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podStartSLOduration=9.266800183 podStartE2EDuration="9.266800183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:40.26639344 +0000 UTC m=+1615.110065071" watchObservedRunningTime="2026-10-30 20:08:40.266800183 +0000 UTC m=+1615.110471814" Oct 30 20:08:40.390535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.390478 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podStartSLOduration=8.390462883 podStartE2EDuration="8.390462883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:40.387272577 +0000 UTC m=+1615.230944207" watchObservedRunningTime="2026-10-30 20:08:40.390462883 +0000 UTC m=+1615.234134512" Oct 30 20:08:40.622046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.621963 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:40.623583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.623554 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:40.623725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.623608 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:40.732158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.732127 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:40.733519 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.733485 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:40.733678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:40.733535 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:41.625877 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:41.621758 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:41.625877 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:41.621817 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:41.741724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:41.740941 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:41.741724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:41.741000 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:41.821732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:41.818715 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:42.622745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:42.622663 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:42.622950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:42.622745 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:42.733202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:42.732998 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:42.733202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:42.733054 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:42.892782 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:08:42.891050 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:08:43.414655 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:43.414584 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:43.622027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:43.621997 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:43.622274 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:43.622248 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:43.733251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:43.733164 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:43.733251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:43.733213 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:44.622417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:44.622378 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:44.622626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:44.622435 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:44.732346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:44.732312 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:44.732514 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:44.732366 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:45.540672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:45.540642 4524 scope.go:122] "RemoveContainer" containerID="818c95e9c1a6b040872248460866df4ed8c89dca21badea782371898f576d73b" Oct 30 20:08:45.622263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:45.622236 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:45.622373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:45.622304 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:45.733047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:45.733011 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:45.733222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:45.733070 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:46.621865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:46.621831 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:08:46.622389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:46.621891 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:08:46.732994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:46.732961 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:08:46.733173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:46.733016 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:08:47.384721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.379783 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:47.692089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.692022 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:47.702609 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.702575 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:08:47.709857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.709837 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.770654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.770617 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwg7\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.774927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.774400 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:08:47.775143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.774900 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775284 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775424 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775550 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775644 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775824 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.775941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.775929 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.776027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.776016 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.801758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.799038 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:47.801758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.800264 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:47.801758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.801607 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:08:47.846814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.846773 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:47.847584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.847566 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:47.849776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.849759 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:08:47.877920 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.877893 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.877939 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.877975 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878145 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878179 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878272 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878300 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878327 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878360 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878428 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878572 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878517 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878572 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878562 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwg7\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.878685 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.878644 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.879048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.879027 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.880652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.880633 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.880771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.880754 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.941139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.941113 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:47.941264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:47.941232 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwg7\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:48.012661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.012598 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:08:48.018113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.018065 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.034258 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.034230 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:48.044676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.044647 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:08:48.049523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.049496 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082648 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082715 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082749 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8xg\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082850 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082905 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082961 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.082985 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.083183 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.083254 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.086416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.084186 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:08:48.144775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.142730 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:08:48.185802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.184882 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.186142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186100 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.186330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186312 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.186492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186472 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.186644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186627 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.186802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186785 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4dv\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.186968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.186950 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.187113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187098 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.187265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187250 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.187399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187384 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.187535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187519 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.187653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187637 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.187800 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8xg\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.187966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.187947 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.188150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.188133 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.188313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.188298 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.188440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.188426 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.188567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.188553 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.189464 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.189441 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.190249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.190224 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.190249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.190240 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.190463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.190258 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.190644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.190623 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.192219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.192189 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.192930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.192890 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.255732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.255680 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.256010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.255986 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8xg\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.290489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290403 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290443 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290471 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290509 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290546 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4dv\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290599 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290674 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.290812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290795 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290914 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290929 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.290968 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.291266 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.291297 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.291466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.291450 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.293012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.292988 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.293098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.293025 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.334307 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.334284 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:48.353102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.353068 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4dv\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.353591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.353570 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.387200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.386890 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:48.456129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.456095 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:08:48.457094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.457054 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.457208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.457134 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.457273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.457203 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.723957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.723928 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:08:48.724861 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:08:48.724838 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb227a_c86c_4838_80ef_756d2af0ff40.slice/crio-d102b0d56625cd619d63932381c52da13fbba0b4738e9b8874779a4cab172881 WatchSource:0}: Error finding container d102b0d56625cd619d63932381c52da13fbba0b4738e9b8874779a4cab172881: Status 404 returned error can't find the container with id d102b0d56625cd619d63932381c52da13fbba0b4738e9b8874779a4cab172881 Oct 30 20:08:48.727976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.727941 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.728065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.728022 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.728107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.728060 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.801446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.801415 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:08:48.802515 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:08:48.802484 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47da3bba_58fa_47a1_b7b5_cc9d2462c22c.slice/crio-cad14077215af53d73348a3898704228a8bad66d3d4d0fb24f60ebdb97381b4c WatchSource:0}: Error finding container cad14077215af53d73348a3898704228a8bad66d3d4d0fb24f60ebdb97381b4c: Status 404 returned error can't find the container with id cad14077215af53d73348a3898704228a8bad66d3d4d0fb24f60ebdb97381b4c Oct 30 20:08:48.805096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.805065 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.805182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.805129 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:48.805182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:48.805159 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:49.307924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.307893 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" event={"ID":"47da3bba-58fa-47a1-b7b5-cc9d2462c22c","Type":"ContainerStarted","Data":"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d"} Oct 30 20:08:49.308086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.307936 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" event={"ID":"47da3bba-58fa-47a1-b7b5-cc9d2462c22c","Type":"ContainerStarted","Data":"cad14077215af53d73348a3898704228a8bad66d3d4d0fb24f60ebdb97381b4c"} Oct 30 20:08:49.308086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.307954 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" event={"ID":"eba67f32-faaa-4efe-a13f-e68867169e7c","Type":"ContainerStarted","Data":"8c1eeb668ff0a034ed1d919b4d261fe6ca58e5605169016439e8ff0075e2578b"} Oct 30 20:08:49.308086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.307970 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" event={"ID":"eba67f32-faaa-4efe-a13f-e68867169e7c","Type":"ContainerStarted","Data":"38fa971a43c36675b8a999a8dd602ce36e81653986cd0496ec4776ca8a82eb7f"} Oct 30 20:08:49.308086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.307986 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" event={"ID":"b6cb227a-c86c-4838-80ef-756d2af0ff40","Type":"ContainerStarted","Data":"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086"} Oct 30 20:08:49.308086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.308008 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" event={"ID":"b6cb227a-c86c-4838-80ef-756d2af0ff40","Type":"ContainerStarted","Data":"d102b0d56625cd619d63932381c52da13fbba0b4738e9b8874779a4cab172881"} Oct 30 20:08:49.334571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.334446 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:49.335997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.335960 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:49.336170 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.336016 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:49.389350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.389315 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:49.390769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.390736 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:49.390997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.390972 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:49.472722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.472192 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podStartSLOduration=2.47217109 podStartE2EDuration="2.47217109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:49.463175885 +0000 UTC m=+1624.306847515" watchObservedRunningTime="2026-10-30 20:08:49.47217109 +0000 UTC m=+1624.315842721" Oct 30 20:08:49.588781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.588233 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podStartSLOduration=2.588214927 podStartE2EDuration="2.588214927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:49.587598751 +0000 UTC m=+1624.431270406" watchObservedRunningTime="2026-10-30 20:08:49.588214927 +0000 UTC m=+1624.431886558" Oct 30 20:08:49.728252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:49.728188 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podStartSLOduration=2.728168051 podStartE2EDuration="2.728168051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:49.709368425 +0000 UTC m=+1624.553040055" watchObservedRunningTime="2026-10-30 20:08:49.728168051 +0000 UTC m=+1624.571839682" Oct 30 20:08:50.034666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.034626 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:50.036733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.036690 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:50.036823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.036771 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:50.336649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.336132 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:50.336649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.336207 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:50.391136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.391099 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:50.391420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:50.391389 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:51.035241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.035202 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:51.035642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.035255 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:51.335593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.335503 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:51.335876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.335844 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:51.388724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.388333 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:51.388724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:51.388391 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:52.035290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.035253 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:52.035689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.035305 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:52.334815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.334731 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:52.335041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.335007 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:52.408724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.391643 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:52.408724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.391734 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:52.605972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:52.605883 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:53.038138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.038092 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:53.038959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.038923 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:53.335964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.335275 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:53.335964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.335354 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:53.398790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.398752 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:53.400974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:53.400938 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:54.036042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.035227 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:54.038689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.036240 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:54.336384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.336299 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:54.336384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.336350 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:54.403748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.397135 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:54.403748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:54.397201 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:55.034686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.034652 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:08:55.034876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.034720 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 20:08:55.335391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.335304 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:08:55.335391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.335367 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:08:55.388111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.388083 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:08:55.388311 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:55.388284 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 20:08:56.531949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.531915 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:56.533341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.533318 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:56.534844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.534826 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:08:56.749690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.749661 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:56.750275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.750230 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:56.751053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.751033 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:08:56.984954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.984921 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:56.985540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.985504 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:56.986294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:56.986261 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:08:57.864886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:08:57.857271 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:02.820215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:02.820177 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.128276 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.128203 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:09:03.132259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.132241 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.208786 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.208756 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.208915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.208801 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.208964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.208941 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.208982 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.209039 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.209066 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9rq\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.209091 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.209122 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.209278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.209260 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.239097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.239072 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:09:03.310998 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.310965 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311069 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311098 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311143 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311167 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311229 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311441 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9rq\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311469 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311547 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311582 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311749 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311693 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.311884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311818 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.312002 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.311971 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.313646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.313616 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.313887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.313871 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.373143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.373114 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.373438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.373418 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9rq\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.447036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.447009 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:03.899978 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:09:03.899945 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d64291_d039_4acc_9240_6182267c1848.slice/crio-8774f70011db1553b7002af7c307f60a4efee0f27896b16d3ef272e480d8caf4 WatchSource:0}: Error finding container 8774f70011db1553b7002af7c307f60a4efee0f27896b16d3ef272e480d8caf4: Status 404 returned error can't find the container with id 8774f70011db1553b7002af7c307f60a4efee0f27896b16d3ef272e480d8caf4 Oct 30 20:09:03.900294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.900130 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:09:03.902773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.902741 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:03.902894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.902843 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:03.902894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:03.902887 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:04.566723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:04.540859 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" event={"ID":"25d64291-d039-4acc-9240-6182267c1848","Type":"ContainerStarted","Data":"2a60d3cc3c25f55c65f1efc1d330c3c19d28257b96fa759826ab384dcd43a3e2"} Oct 30 20:09:04.566723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:04.540928 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" event={"ID":"25d64291-d039-4acc-9240-6182267c1848","Type":"ContainerStarted","Data":"8774f70011db1553b7002af7c307f60a4efee0f27896b16d3ef272e480d8caf4"} Oct 30 20:09:04.685802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:04.685744 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podStartSLOduration=1.685722775 podStartE2EDuration="1.685722775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:04.671024803 +0000 UTC m=+1639.514696433" watchObservedRunningTime="2026-10-30 20:09:04.685722775 +0000 UTC m=+1639.529394406" Oct 30 20:09:05.461726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:05.459785 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:05.481844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:05.481805 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:05.482020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:05.481867 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:06.450052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:06.450016 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:06.450336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:06.450304 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:07.452726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:07.449993 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:07.452726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:07.450061 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:07.843414 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:07.843363 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:08.448719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.448073 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:08.448719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.448136 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:08.755772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.754083 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9146-d7fc9a"] Oct 30 20:09:08.811028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.810887 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9146-d7fc9a"] Oct 30 20:09:08.811335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.811319 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:08.910723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:08.888367 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nts8c\"" Oct 30 20:09:09.670722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.665391 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:09.706752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.688184 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5gv\" (UniqueName: \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\") pod \"test-curl-9146-d7fc9a\" (UID: \"d90b868e-ad58-401a-a441-f2145e64a757\") " pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:09.773822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.773777 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:09:09.882766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.882669 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:09.883053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.883022 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:09.956784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:09.955097 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5gv\" (UniqueName: \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\") pod \"test-curl-9146-d7fc9a\" (UID: \"d90b868e-ad58-401a-a441-f2145e64a757\") " pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:10.118119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.117319 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5gv\" (UniqueName: \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\") pod \"test-curl-9146-d7fc9a\" (UID: \"d90b868e-ad58-401a-a441-f2145e64a757\") " pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:10.202161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.201857 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:10.467730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.460742 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:10.467730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.460818 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:10.662724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.643850 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:10.686726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.686142 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:10.686726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.686584 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:09:10.784725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.783808 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:10.784725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.784231 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:09:10.784725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:10.784556 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:09:11.199186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.196352 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:11.199186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.196414 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:11.509254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.498050 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:11.509254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.498155 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:11.673229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.673186 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:11.719785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.693546 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:11.746245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.720375 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:09:11.792670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.784776 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:11.856094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.856045 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:09:11.859892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:11.859850 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:09:12.025164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.025135 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9146-d7fc9a"] Oct 30 20:09:12.451722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.451153 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:12.451722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.451217 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.612978 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.612832 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:12.612978 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.612899 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.615742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.615555 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:12.615742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.615613 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.619043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.619016 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:12.619222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.619199 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.754783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.742816 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.766722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.760158 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:12.813722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.813511 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:12.813722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.813590 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:12.852725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.841414 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:09:12.892263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:12.874778 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.031386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.019606 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:09:13.066720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.066290 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:13.549822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.541356 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" podUID="6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.599488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.581963 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mnfw4" podUID="6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.708733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.699293 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:13.708733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.699360 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.761724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.752877 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.779723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.774042 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.823779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.811568 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:13.823779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.811627 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:13.890977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.857356 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.903763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.901500 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:13.922018 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:13.921039 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:09:13.922018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:13.921496 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:14.026172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.021233 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:14.026172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.021291 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.459776 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:14.452501 4524 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.064s" Oct 30 20:09:14.459776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.452593 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9146-d7fc9a" event={"ID":"d90b868e-ad58-401a-a441-f2145e64a757","Type":"ContainerStarted","Data":"229e63838b593ba9b6c135bdab931672cdd281e21bf8f6608893481b9728c3ff"} Oct 30 20:09:14.547743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.513827 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:14.547743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.513890 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:14.709498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.709461 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:14.709855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.709767 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.771319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.769954 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.791854 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.725069 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.836109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.836056 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.917896 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.909987 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 20:09:14.946724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.921586 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:14.960471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:14.926976 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:15.476312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.475936 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:09:15.476933 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.476807 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:09:15.976271 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.976227 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:15.978745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.977860 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:15.993440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.993404 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:15.996807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.996788 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:09:15.997007 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:15.996995 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:09:16.227747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:16.227633 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:09:16.236729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:16.233115 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:09:16.244535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:16.244505 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:16.545194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:16.545082 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:16.812781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:16.794638 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:17.136910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:17.136798 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:17.204991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:17.203105 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:17.204991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:17.204838 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:17.206807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:17.206759 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:09:17.834120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:17.833946 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.327733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:18.323608 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.864722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:18.861837 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:20.936940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:20.936900 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9146-d7fc9a" event={"ID":"d90b868e-ad58-401a-a441-f2145e64a757","Type":"ContainerStarted","Data":"eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55"} Oct 30 20:09:21.063778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:21.063715 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9146-d7fc9a" podStartSLOduration=5.741628481 podStartE2EDuration="13.063682194s" totalImagesPullingTime="7.322053713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:21.059891544 +0000 UTC m=+1655.903563174" watchObservedRunningTime="2026-10-30 20:09:21.063682194 +0000 UTC m=+1655.907353824" Oct 30 20:09:21.301657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:21.301581 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:21.627429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:21.627337 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.036244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.036204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.319959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.319605 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.581181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.581078 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.730622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.730590 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9146-d7fc9a"] Oct 30 20:09:22.768819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.766039 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9146-d7fc9a"] Oct 30 20:09:22.851984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.851896 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.950350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:22.950305 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9146-d7fc9a" podUID="d90b868e-ad58-401a-a441-f2145e64a757" containerName="test-curl-9146-d7fc9a" containerID="cri-o://eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55" gracePeriod=2 Oct 30 20:09:23.002325 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:23.002285 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:23.106775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.105738 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:09:23.113735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.112017 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d90b868e-ad58-401a-a441-f2145e64a757" containerName="test-curl-9146-d7fc9a" Oct 30 20:09:23.113735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.112044 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90b868e-ad58-401a-a441-f2145e64a757" containerName="test-curl-9146-d7fc9a" Oct 30 20:09:23.113735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.112303 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90b868e-ad58-401a-a441-f2145e64a757" containerName="test-curl-9146-d7fc9a" Oct 30 20:09:23.123733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.117897 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.247755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.221479 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7ca5342-openshift-default-dockercfg-j6r4g\"" Oct 30 20:09:23.272743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.267248 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.353741 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.353814 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.353947 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354000 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxg7\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354038 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354087 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354113 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354158 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.356688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.354222 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.498348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.467831 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.508853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.508816 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.528975 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529059 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxg7\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529108 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529171 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529194 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529234 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529318 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.529975 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:23.497400 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.530244 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.530742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.509450 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.531474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.530814 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.549269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.549233 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.550538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.550497 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.570451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.570417 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.579871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.579661 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:23.585355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.581087 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:09:23.625726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.622030 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxg7\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.665814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.665778 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\") pod \"e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:23.927722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.925831 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:23.942721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:23.941004 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:24.150957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.150924 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:24.460727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.455371 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:24.588723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.586206 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:09:24.588723 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:09:24.588545 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5d3de7_b4dc_4df5_8f8e_9f6109bc42f6.slice/crio-5d115d42c628399d4456fdb6a9910e3a9bc34968b5167d94506dd12c39634276 WatchSource:0}: Error finding container 5d115d42c628399d4456fdb6a9910e3a9bc34968b5167d94506dd12c39634276: Status 404 returned error can't find the container with id 5d115d42c628399d4456fdb6a9910e3a9bc34968b5167d94506dd12c39634276 Oct 30 20:09:24.704723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.704278 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:24.704723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.704390 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:24.704723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.704435 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:09:24.820633 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:24.820598 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:25.033947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.033144 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerStarted","Data":"5d115d42c628399d4456fdb6a9910e3a9bc34968b5167d94506dd12c39634276"} Oct 30 20:09:25.113967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.113489 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:25.332628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.332607 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:25.384533 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:25.384494 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:25.406296 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.406262 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:25.417066 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.417024 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\" (UniqueName: \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\") pod \"d90b868e-ad58-401a-a441-f2145e64a757\" (UID: \"d90b868e-ad58-401a-a441-f2145e64a757\") " Oct 30 20:09:25.419113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.419086 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv" pod "d90b868e-ad58-401a-a441-f2145e64a757" (UID: "d90b868e-ad58-401a-a441-f2145e64a757"). InnerVolumeSpecName "kube-api-access-7q5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.518857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.518824 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q5gv\" (UniqueName: \"kubernetes.io/projected/d90b868e-ad58-401a-a441-f2145e64a757-kube-api-access-7q5gv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:09:25.682792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:25.682760 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:26.041521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.041437 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerStarted","Data":"21ace7a70aee672c1584e88f359c7221e3cf86e7f177511de28182a943b8d351"} Oct 30 20:09:26.042633 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.042603 4524 generic.go:356] "Generic (PLEG): container finished" podID="d90b868e-ad58-401a-a441-f2145e64a757" containerID="eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55" exitCode=137 Oct 30 20:09:26.042831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.042644 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:26.042831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.042691 4524 scope.go:122] "RemoveContainer" containerID="eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55" Oct 30 20:09:26.054109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.054078 4524 scope.go:122] "RemoveContainer" containerID="eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55" Oct 30 20:09:26.054401 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:26.054377 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55\": container with ID starting with eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55 not found: ID does not exist" containerID="eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55" Oct 30 20:09:26.054472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.054415 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55"} err="failed to get container status \"eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55\": rpc error: code = NotFound desc = could not find container \"eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55\": container with ID starting with eb31381e513f9e0e249a3b03ce25bfa9ec534d38449559fb6e52fc653ffded55 not found: ID does not exist" Oct 30 20:09:26.219811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.219749 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podStartSLOduration=4.219730017 podStartE2EDuration="4.219730017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:26.215993238 +0000 UTC m=+1661.059664865" watchObservedRunningTime="2026-10-30 20:09:26.219730017 +0000 UTC m=+1661.063401643" Oct 30 20:09:26.267537 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:26.267502 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:26.318837 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:26.318759 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:26.419013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.412204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:26.948750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.945671 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:26.972730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:26.969002 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:27.020527 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:27.020486 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:27.049518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:27.049491 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:27.050332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:27.050311 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:09:27.090666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:27.090611 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:27.437720 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:27.427077 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9146-d7fc9a\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="d90b868e-ad58-401a-a441-f2145e64a757" pod="opendatahub/test-curl-9146-d7fc9a" Oct 30 20:09:27.574619 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:27.574527 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:27.575254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:27.575214 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90b868e-ad58-401a-a441-f2145e64a757" path="/var/lib/kubelet/pods/d90b868e-ad58-401a-a441-f2145e64a757/volumes" Oct 30 20:09:28.711349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:28.711317 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:28.987617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:28.987523 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:34.332724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:34.329295 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:35.012761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:35.012730 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:37.935734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:37.935675 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.181130 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:38.181096 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.557931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:38.557896 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:39.742362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:39.742317 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:41.601025 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:41.600963 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:42.788536 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:09:42.788497 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:09:43.753809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:43.753775 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:44.574440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:44.574385 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:45.090724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:45.090675 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:45.440863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:45.440829 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:45.661786 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:45.661757 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:45.892368 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:45.892344 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:49.248724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:49.242783 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:49.364725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:49.363999 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:49.764734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:49.756801 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:49.889359 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:49.879802 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:09:50.275755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.244836 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:50.328725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.284008 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:09:50.442759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.382341 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:50.506436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.506400 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:09:50.774718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.766327 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:09:50.774718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.768138 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:09:50.885731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.883454 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:09:50.899727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:50.898055 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:09:51.278080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:51.278028 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:51.512505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:51.512465 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:51.788734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:51.781784 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:09:51.884891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:51.884847 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:52.436161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:52.435890 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:52.536138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:52.536090 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:09:52.829781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:52.825673 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:52.903452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:52.903399 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:53.448302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:53.448258 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:53.591803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:53.591691 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:09:53.864267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:53.864174 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:09:53.929688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:53.929638 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.102912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.102879 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:54.432097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.432060 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:54.455804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.455752 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.629724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.622309 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.631632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.629251 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.654630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.654592 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:54.654915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.654887 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.869768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.868854 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:54.904150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.895065 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 20:09:54.939777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:54.936670 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:55.013058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.005252 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:55.315189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.314980 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:55.468489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.468448 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:09:55.664573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.632393 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:55.897899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.897847 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:55.937979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.937838 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:55.955787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:55.955756 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:09:56.003728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:56.003677 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:09:56.084770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:56.084744 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:09:56.161920 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:09:56.161881 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:10:00.180729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:00.179739 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:00.797257 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:00.797224 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:01.267450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.267410 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:10:01.274035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.274012 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.329100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.329060 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-770dfe-openshift-default-dockercfg-8vr9k\"" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375181 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375289 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375389 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375412 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375470 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375527 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375602 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375679 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7dq\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.381724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.375815 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515195 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515298 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7dq\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515391 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515440 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515499 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515570 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515586 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515635 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.515676 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.516066 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.516251 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.516689 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.540771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.517021 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.551560 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.544966 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:01.562949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.562912 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.592510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.592480 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:10:01.626802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.626723 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7dq\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.635853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.635582 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.637938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.637910 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.641266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.641235 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\") pod \"e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:01.889615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:01.889544 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:02.184124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.175021 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:02.399954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.399924 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:10:02.400422 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:10:02.400400 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5f34ff_8ed9_4a77_a66a_2ce8f71b9347.slice/crio-f04bf55ffc9248c065d0d5101f61e537aeef2fa8205b7c359e42d9392302c67f WatchSource:0}: Error finding container f04bf55ffc9248c065d0d5101f61e537aeef2fa8205b7c359e42d9392302c67f: Status 404 returned error can't find the container with id f04bf55ffc9248c065d0d5101f61e537aeef2fa8205b7c359e42d9392302c67f Oct 30 20:10:02.401736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.401693 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:02.403139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.403109 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:02.403209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.403175 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:02.403209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.403206 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:02.779641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.779557 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerStarted","Data":"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09"} Oct 30 20:10:02.779854 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.779652 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerStarted","Data":"f04bf55ffc9248c065d0d5101f61e537aeef2fa8205b7c359e42d9392302c67f"} Oct 30 20:10:02.890457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.890429 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:02.902734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:02.902650 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podStartSLOduration=1.902636856 podStartE2EDuration="1.902636856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:02.901537632 +0000 UTC m=+1697.745209261" watchObservedRunningTime="2026-10-30 20:10:02.902636856 +0000 UTC m=+1697.746308484" Oct 30 20:10:03.896100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:03.896074 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:04.795956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:04.795923 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:04.796924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:04.796905 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:10:05.586173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:05.586069 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:05.821783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:05.821751 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:06.086024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:06.085982 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:08.805375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.805345 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:10:08.809777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.809757 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.855018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.854988 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:10:08.859034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.859015 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.875659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.875633 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\"" Oct 30 20:10:08.914461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.914428 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:10:08.930110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930079 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930121 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930273 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.930406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930345 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.930487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930461 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.930547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930507 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930632 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.930803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930691 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930743 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzkq\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930811 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.930926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930882 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.930926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930913 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q7m\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.931089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.930973 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.931148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.931085 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.931220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.931199 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:08.931278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.931238 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.931362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.931342 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.931422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.931392 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:08.968163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:08.965504 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:10:09.032356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032326 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032362 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032385 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzkq\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032407 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032441 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032457 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q7m\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032495 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032557 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032638 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032667 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032756 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032802 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032827 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.032856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.032847 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033011 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033087 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033090 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033178 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033248 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033250 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033290 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033325 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033612 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033676 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.033729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033715 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.033852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.033827 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.034095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.034073 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.034203 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.034094 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.035401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.035385 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.035525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.035503 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.035684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.035669 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.036034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.036018 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.093662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.093598 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q7m\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.098045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.098015 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.100371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.100293 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzkq\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dllfw\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.101003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.100978 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xpbtq\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.125033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.125010 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:09.173071 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.173042 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:09.301752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.301693 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:09.608500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.608437 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:09.656744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.653829 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:10:09.684604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.684566 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:09.684768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.684655 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:09.684768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.684718 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:09.694103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.694072 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:10:09.696167 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:10:09.696140 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad1b637_ba01_4463_965a_7721df84b194.slice/crio-8ce3195eb3c5b203c2b71cec742874b27bfbb39af82afa33fa1064b02e16a8c4 WatchSource:0}: Error finding container 8ce3195eb3c5b203c2b71cec742874b27bfbb39af82afa33fa1064b02e16a8c4: Status 404 returned error can't find the container with id 8ce3195eb3c5b203c2b71cec742874b27bfbb39af82afa33fa1064b02e16a8c4 Oct 30 20:10:09.698964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.698933 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:09.699062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.699011 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:09.699124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:09.699082 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:10.005848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:10.000761 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" event={"ID":"d6c1367f-7dc1-45c3-a6dc-49a22793714c","Type":"ContainerStarted","Data":"2011876388f0e77e41fca44b12cc2d5886be760892346a56a1152289519a3b3d"} Oct 30 20:10:10.136097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:10.136059 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" event={"ID":"4ad1b637-ba01-4463-965a-7721df84b194","Type":"ContainerStarted","Data":"8ce3195eb3c5b203c2b71cec742874b27bfbb39af82afa33fa1064b02e16a8c4"} Oct 30 20:10:11.073453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.073396 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:11.275925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.275874 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" event={"ID":"d6c1367f-7dc1-45c3-a6dc-49a22793714c","Type":"ContainerStarted","Data":"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851"} Oct 30 20:10:11.292426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.292366 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" event={"ID":"4ad1b637-ba01-4463-965a-7721df84b194","Type":"ContainerStarted","Data":"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445"} Oct 30 20:10:11.357284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.344282 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:11.517375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.517292 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podStartSLOduration=3.517268909 podStartE2EDuration="3.517268909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:11.473383899 +0000 UTC m=+1706.317055526" watchObservedRunningTime="2026-10-30 20:10:11.517268909 +0000 UTC m=+1706.360940541" Oct 30 20:10:11.661723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.660633 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podStartSLOduration=3.660608988 podStartE2EDuration="3.660608988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:11.623244297 +0000 UTC m=+1706.466915938" watchObservedRunningTime="2026-10-30 20:10:11.660608988 +0000 UTC m=+1706.504280617" Oct 30 20:10:11.714752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:11.714231 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:12.125662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.125590 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:12.126894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.126868 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:12.126992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.126926 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:12.175172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.175132 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:12.175429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.175299 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:12.175429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:12.175357 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:12.839832 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:12.839789 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:10:13.125597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:13.125501 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:13.125597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:13.125565 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:13.175032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:13.173428 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:13.175032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:13.173492 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:14.126008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:14.125962 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:14.126423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:14.126028 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:14.179090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:14.179056 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:14.179365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:14.179335 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:15.126185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:15.126145 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:15.126789 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:15.126755 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:15.185724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:15.174007 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:15.185724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:15.174065 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:16.125654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.125622 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:16.125863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.125672 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:16.181871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.181836 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:16.182958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.182929 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:16.417427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.417396 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:10:16.422011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.421988 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.525741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.518531 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.525741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.522777 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.525741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.523122 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.525741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.523160 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64fs\" (UniqueName: \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.560995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.560961 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-80e4d90e-serving-cert\"" Oct 30 20:10:16.625584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.625547 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.625935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.625914 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.626501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.626480 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.626659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.626645 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64fs\" (UniqueName: \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.626908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.626875 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.627096 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:16.626052 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-serving-cert: secret "maas-api-e2e-derive-80e4d90e-serving-cert" not found Oct 30 20:10:16.627180 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:16.627099 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls podName:e9c2745f-5a87-4d9d-b92e-f07c71dddc86 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:17.127078558 +0000 UTC m=+1711.970750184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls") pod "maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86") : secret "maas-api-e2e-derive-80e4d90e-serving-cert" not found Oct 30 20:10:16.632453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.632430 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.633730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.632849 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:10:16.726389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.726306 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:16.772437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.763543 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64fs\" (UniqueName: \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:16.951612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:16.951574 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:17.128056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.127955 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:17.133737 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:17.128446 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-serving-cert: secret "maas-api-e2e-derive-80e4d90e-serving-cert" not found Oct 30 20:10:17.133737 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:17.128516 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls podName:e9c2745f-5a87-4d9d-b92e-f07c71dddc86 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:18.12849834 +0000 UTC m=+1712.972169965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls") pod "maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86") : secret "maas-api-e2e-derive-80e4d90e-serving-cert" not found Oct 30 20:10:17.133737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.129208 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:17.133737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.129250 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:17.175230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.175152 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:17.175518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.175219 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:17.417412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:17.417377 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:18.125995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.125960 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:10:18.126152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.126029 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:10:18.145981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.145928 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:18.149172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.149141 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:18.174076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.174041 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 20:10:18.174195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.174116 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 20:10:18.239567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.239536 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:18.652925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:18.651762 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:10:18.656560 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:10:18.656529 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c2745f_5a87_4d9d_b92e_f07c71dddc86.slice/crio-6a1e77481da50af290e36abb0689b5ca49cc1c88b79e5b55be32e18ba3c2faa6 WatchSource:0}: Error finding container 6a1e77481da50af290e36abb0689b5ca49cc1c88b79e5b55be32e18ba3c2faa6: Status 404 returned error can't find the container with id 6a1e77481da50af290e36abb0689b5ca49cc1c88b79e5b55be32e18ba3c2faa6 Oct 30 20:10:19.666722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:19.666487 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" event={"ID":"e9c2745f-5a87-4d9d-b92e-f07c71dddc86","Type":"ContainerStarted","Data":"ae726c8935fb286067ed9608ab66512f010d1ca53e4333e6b089da2669d7fa03"} Oct 30 20:10:19.666722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:19.666554 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" event={"ID":"e9c2745f-5a87-4d9d-b92e-f07c71dddc86","Type":"ContainerStarted","Data":"6a1e77481da50af290e36abb0689b5ca49cc1c88b79e5b55be32e18ba3c2faa6"} Oct 30 20:10:19.668745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:19.667834 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:19.798775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:19.793434 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" podStartSLOduration=3.008384784 podStartE2EDuration="3.79341405s" totalImagesPullingTime="785.029266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:19.788062265 +0000 UTC m=+1714.631733895" watchObservedRunningTime="2026-10-30 20:10:19.79341405 +0000 UTC m=+1714.637085679" Oct 30 20:10:20.126534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.126443 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dllfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.126922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.126893 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.259891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.259827 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.373509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.373016 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.569182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.557843 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:20.569182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.562522 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:20.569182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.567350 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:10:20.601145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.601110 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:20.604033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.603809 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:20.605217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:20.605198 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:10:22.185804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:22.173245 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:23.933677 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.933640 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:10:23.938308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.938287 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979151 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979189 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57k7q\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979257 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979300 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979324 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979444 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979471 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979616 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:23.979726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:23.979695 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.042737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.041620 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:10:24.080762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080735 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.080910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080787 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.080910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080814 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57k7q\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.080910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080840 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.080910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080903 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.080938 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081074 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081107 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081284 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081354 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081469 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081447 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081692 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081739 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.081971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.081927 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.083467 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.083445 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.083646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.083624 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.141930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.141906 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.142099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.142080 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57k7q\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9rckh\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.253391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.253302 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:24.889721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.882493 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:24.889721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.882584 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:24.889721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.882625 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:24.899725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:24.894951 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:10:25.735305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:25.735268 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" event={"ID":"0807c7d0-ad5b-4740-bfee-03ac643226e9","Type":"ContainerStarted","Data":"b53f3225fe2c648d8865199106a1eb4ff59acc525f0bbeaa016598aaae09ac6f"} Oct 30 20:10:25.735798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:25.735322 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" event={"ID":"0807c7d0-ad5b-4740-bfee-03ac643226e9","Type":"ContainerStarted","Data":"86d6c38a52999f922551407b3412fc67378af6a7906adcbab00b5c32895ddde6"} Oct 30 20:10:25.859978 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:25.859912 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podStartSLOduration=2.859889791 podStartE2EDuration="2.859889791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:25.857424448 +0000 UTC m=+1720.701096077" watchObservedRunningTime="2026-10-30 20:10:25.859889791 +0000 UTC m=+1720.703561421" Oct 30 20:10:26.254888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:26.254847 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:26.282895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:26.282847 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:26.283201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:26.283152 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:26.702108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:26.700868 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:10:27.364722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:27.338690 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:27.364722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:27.338766 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:27.980721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:27.956474 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:28.225294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:28.225256 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:28.315812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:28.313190 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:28.315812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:28.313258 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:29.259571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:29.259103 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:29.259571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:29.259169 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:30.256725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:30.254057 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:30.256725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:30.254117 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:31.269852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:31.269815 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:31.270457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:31.270422 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:32.254271 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:32.254233 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:32.254448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:32.254297 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:33.254446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:33.254409 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9rckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:10:33.254951 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:33.254485 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 20:10:33.375731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:33.375642 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:34.502969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:34.502934 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:34.503467 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:34.503205 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:34.504564 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:34.504537 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:10:38.261722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:38.258665 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:42.787552 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:10:42.787495 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:10:43.274452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:43.269656 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:44.112210 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:44.112168 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.163077 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:45.163040 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.378116 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:45.378071 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:48.157609 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:48.157538 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.239907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:53.239874 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.091924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:58.091825 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:58.091924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:58.091897 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:58.153848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:58.153751 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.132583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.132554 4524 generic.go:356] "Generic (PLEG): container finished" podID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerID="21ace7a70aee672c1584e88f359c7221e3cf86e7f177511de28182a943b8d351" exitCode=0 Oct 30 20:10:59.133048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.132617 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerDied","Data":"21ace7a70aee672c1584e88f359c7221e3cf86e7f177511de28182a943b8d351"} Oct 30 20:10:59.133112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.133092 4524 scope.go:122] "RemoveContainer" containerID="21ace7a70aee672c1584e88f359c7221e3cf86e7f177511de28182a943b8d351" Oct 30 20:10:59.134836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.134804 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:59.134938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.134874 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:59.134938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:10:59.134907 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:00.140949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:00.140917 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerStarted","Data":"566a0437c1d0a766c6e5f872e0f1819d76150572922010f18fc8a516d4eaddec"} Oct 30 20:11:00.942358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:00.942322 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:11:00.942524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:00.942476 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:11:00.942524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:00.942511 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:11:01.942063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:01.942027 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:11:01.942552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:01.942100 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:11:02.941895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:02.941861 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:11:02.942090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:02.941916 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:11:03.020993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:03.020879 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.259584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:03.259497 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.942689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:03.942654 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:11:03.943080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:03.942716 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:11:03.960118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:03.960083 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:04.957727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:04.947776 4524 patch_prober.go:29] interesting pod/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:11:04.957727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:04.947856 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:11:05.216746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:05.216659 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.476045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:05.475962 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.946537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:05.946506 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:11:05.946946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:05.946930 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:11:05.947676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:05.947660 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:11:08.453904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:08.453523 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.789027 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:11:12.788926 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:11:13.370724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:13.360036 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:16.173242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:16.173203 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:17.949995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:17.946488 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:19.740108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:19.740073 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.032179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:20.032068 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.246797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:20.246758 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.478723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:22.475433 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.660420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:23.660359 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.218731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:25.218665 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.508551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:25.508384 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.947834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:25.947797 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.271038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:27.271002 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.127286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:29.127235 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.829811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:29.829777 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.207719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:31.207266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.440305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:31.440267 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.058642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:33.058606 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.294147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:33.294089 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.574160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:33.574111 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.257390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:34.257308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.729720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:34.726444 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.030333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:35.027058 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.566164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:36.566102 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.807044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:36.806997 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:38.599818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:38.599786 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:38.885446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:38.885358 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.398721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:40.390290 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.949411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:41.949373 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.208196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.185281 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.723832 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.720451 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:11:42.799942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.799907 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:11:42.801787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.801129 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:42.907912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.907881 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7961ed\"" Oct 30 20:11:42.908200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.908183 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7961ed-dockercfg-7ngmj\"" Oct 30 20:11:42.933727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.931791 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8zh\" (UniqueName: \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:42.933727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:42.932115 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.019345 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:11:43.019252 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:11:43.044046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.044005 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.074651 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.074614 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8zh\" (UniqueName: \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.085995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.049081 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.184958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.178434 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8zh\" (UniqueName: \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\") pod \"payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.223253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.223218 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:43.610439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.610414 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:11:43.611086 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:11:43.611062 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4372de_29fa_4564_b59c_6c6ff834d752.slice/crio-f6c67a81bb3f2e4726c59a57a2546b41b6268bd9e26962bea5be75bbaea442c1 WatchSource:0}: Error finding container f6c67a81bb3f2e4726c59a57a2546b41b6268bd9e26962bea5be75bbaea442c1: Status 404 returned error can't find the container with id f6c67a81bb3f2e4726c59a57a2546b41b6268bd9e26962bea5be75bbaea442c1 Oct 30 20:11:43.957344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:43.957307 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" event={"ID":"cf4372de-29fa-4564-b59c-6c6ff834d752","Type":"ContainerStarted","Data":"f6c67a81bb3f2e4726c59a57a2546b41b6268bd9e26962bea5be75bbaea442c1"} Oct 30 20:11:44.457549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:44.457513 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:45.515721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:45.491723 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:46.984552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:46.984516 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" event={"ID":"cf4372de-29fa-4564-b59c-6c6ff834d752","Type":"ContainerStarted","Data":"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a"} Oct 30 20:11:46.985011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:46.984727 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:11:47.114845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:47.114793 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" podStartSLOduration=2.148712948 podStartE2EDuration="5.114778607s" totalImagesPullingTime="2.966065659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:47.111330271 +0000 UTC m=+1801.955001895" watchObservedRunningTime="2026-10-30 20:11:47.114778607 +0000 UTC m=+1801.958450238" Oct 30 20:11:49.537723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:49.531401 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:49.784783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:49.781754 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.375729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:50.374555 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:55.512226 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:55.512184 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.995814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:11:57.995785 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:12:00.293113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:00.293048 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.188725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:05.183968 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.831530 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.831491 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:12:09.838436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.838418 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.898661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.898629 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.898944 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.898919 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899028 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899094 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899185 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zv52\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899221 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899269 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899408 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.899487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.899443 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:09.944823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:09.944797 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:12:10.001035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.000997 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001045 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001113 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001349 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001417 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001459 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001538 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zv52\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001569 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.001672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.001620 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.002777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.002159 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.002777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.002202 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.002777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.002433 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.002777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.002479 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.003773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.003749 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.005157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.005132 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.006306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.006282 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.077534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.077498 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zv52\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.078843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.078817 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5vkt8\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.182274 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.182237 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:10.426055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.425979 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.690424 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:12:10.690393 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb338c1_09b6_481a_b2a2_5d9302620681.slice/crio-6b55f8e220681d5e88300a78285a039966dae87c2c879744f9fd5806e0c3165e WatchSource:0}: Error finding container 6b55f8e220681d5e88300a78285a039966dae87c2c879744f9fd5806e0c3165e: Status 404 returned error can't find the container with id 6b55f8e220681d5e88300a78285a039966dae87c2c879744f9fd5806e0c3165e Oct 30 20:12:10.691648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.691620 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:12:10.693154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.693123 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:10.693242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.693187 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:10.693242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:10.693216 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:11.326607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:11.326571 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" event={"ID":"3fb338c1-09b6-481a-b2a2-5d9302620681","Type":"ContainerStarted","Data":"266476d022bc6e9d61eca0968b82c610f289d583157932cb4eb7c40e2d171265"} Oct 30 20:12:11.327017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:11.326630 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" event={"ID":"3fb338c1-09b6-481a-b2a2-5d9302620681","Type":"ContainerStarted","Data":"6b55f8e220681d5e88300a78285a039966dae87c2c879744f9fd5806e0c3165e"} Oct 30 20:12:11.500788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:11.498606 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podStartSLOduration=2.498584823 podStartE2EDuration="2.498584823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:11.495321584 +0000 UTC m=+1826.338993226" watchObservedRunningTime="2026-10-30 20:12:11.498584823 +0000 UTC m=+1826.342256456" Oct 30 20:12:12.195903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:12.195848 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:12.197534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:12.197510 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:12.197665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:12.197560 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:12.829628 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:12:12.829591 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:12:13.184205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:13.184172 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:13.184492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:13.184463 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:13.996435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:13.996396 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:14.191722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:14.190990 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:14.191722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:14.191087 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:14.514125 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:14.514093 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.035615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.035576 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:12:15.053653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.053622 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116303 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116477 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116587 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116724 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116766 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116810 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116873 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.116935 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrf98\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.134471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.117105 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.201724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.136940 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:12:15.324820 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.249942 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:15.324820 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.250021 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:15.324820 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.250755 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a2103a-openshift-default-dockercfg-f8mx4\"" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332363 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332463 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332563 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332692 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332752 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332795 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332859 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.332922 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrf98\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.333023 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.333725 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.344208 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.344870 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.345602 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.375775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.345880 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.438192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.416880 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.737175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.737139 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:15.740713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.740662 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:16.047606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.767003 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrf98\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:16.048808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:15.798904 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\") pod \"e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:16.048994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.005244 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:16.058665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.058634 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:16.286111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.273043 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:16.286111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.273115 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:16.308770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.306780 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:16.440663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.440515 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.516336 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:12:16.516285 4524 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.145s" Oct 30 20:12:16.674657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.670551 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.918782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.918751 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:12:16.918994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:16.918969 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" podUID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" containerName="maas-api" containerID="cri-o://ae726c8935fb286067ed9608ab66512f010d1ca53e4333e6b089da2669d7fa03" gracePeriod=30 Oct 30 20:12:17.087576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.086424 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.089109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.089083 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:12:17.103596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.103552 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:17.103824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.103795 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:17.103964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.103944 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:17.210902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.210749 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:17.220544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.220503 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:17.830473 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.830342 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.910551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.909867 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerStarted","Data":"f665cc1332cb2637e1756551b60ab8715d7c374e82aab97101394dd927a94c7f"} Oct 30 20:12:17.912466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.911242 4524 generic.go:356] "Generic (PLEG): container finished" podID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" containerID="ae726c8935fb286067ed9608ab66512f010d1ca53e4333e6b089da2669d7fa03" exitCode=0 Oct 30 20:12:17.912466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.911369 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" event={"ID":"e9c2745f-5a87-4d9d-b92e-f07c71dddc86","Type":"ContainerDied","Data":"ae726c8935fb286067ed9608ab66512f010d1ca53e4333e6b089da2669d7fa03"} Oct 30 20:12:17.914767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:17.914690 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:12:18.068314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.068129 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.105817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.087886 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") pod \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " Oct 30 20:12:18.105817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.088079 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\" (UniqueName: \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\") pod \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " Oct 30 20:12:18.105817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.088356 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\") pod \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " Oct 30 20:12:18.105817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.088414 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\") pod \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\" (UID: \"e9c2745f-5a87-4d9d-b92e-f07c71dddc86\") " Oct 30 20:12:18.146979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.118059 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca" pod "e9c2745f-5a87-4d9d-b92e-f07c71dddc86" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:18.205732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.199938 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs" pod "e9c2745f-5a87-4d9d-b92e-f07c71dddc86" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86"). InnerVolumeSpecName "kube-api-access-t64fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:18.243726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.215146 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:18.243726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.215185 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64fs\" (UniqueName: \"kubernetes.io/projected/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-kube-api-access-t64fs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:18.243726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.227820 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs" pod "e9c2745f-5a87-4d9d-b92e-f07c71dddc86" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:18.244246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.244154 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls" pod "e9c2745f-5a87-4d9d-b92e-f07c71dddc86" (UID: "e9c2745f-5a87-4d9d-b92e-f07c71dddc86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:18.351776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.351009 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:18.351776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.351059 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c2745f-5a87-4d9d-b92e-f07c71dddc86-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:18.371519 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.354901 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.477396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.472234 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:18.477396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.472292 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:18.977722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:18.976524 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:19.002731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.000593 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerStarted","Data":"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c"} Oct 30 20:12:19.006721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.003554 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" event={"ID":"e9c2745f-5a87-4d9d-b92e-f07c71dddc86","Type":"ContainerDied","Data":"6a1e77481da50af290e36abb0689b5ca49cc1c88b79e5b55be32e18ba3c2faa6"} Oct 30 20:12:19.006721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.003609 4524 scope.go:122] "RemoveContainer" containerID="ae726c8935fb286067ed9608ab66512f010d1ca53e4333e6b089da2669d7fa03" Oct 30 20:12:19.006721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.003974 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2" Oct 30 20:12:19.074277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.074179 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:19.075234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.075170 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:12:19.075234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.075224 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:12:19.186631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.183169 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podStartSLOduration=5.183150425 podStartE2EDuration="5.183150425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:19.180344613 +0000 UTC m=+1834.024016242" watchObservedRunningTime="2026-10-30 20:12:19.183150425 +0000 UTC m=+1834.026822054" Oct 30 20:12:19.186631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.183393 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:19.186631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.183444 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:19.308813 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.308736 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:12:19.308813 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.308787 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80e4d90e-78f978dd4b-7ttc2"] Oct 30 20:12:19.406721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.402383 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:19.725725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:19.718207 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.157805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.126033 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:20.217843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.186456 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.218333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.218259 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:20.218425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.218328 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:20.227924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.227897 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:20.243788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.236948 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:12:20.246308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.244266 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:12:20.246308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.245279 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" containerName="maas-api" Oct 30 20:12:20.246308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.245299 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" containerName="maas-api" Oct 30 20:12:20.246308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.245508 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" containerName="maas-api" Oct 30 20:12:20.278533 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.266941 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.342719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.337335 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-577fb85a-openshift-default-dockercfg-g7lxm\"" Oct 30 20:12:20.506774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.506531 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.506774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.506588 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.506774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.506748 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.506781 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.506945 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k996m\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.507016 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.507117 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.507152 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.507238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.507190 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.611914 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612029 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612067 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612103 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612221 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612256 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612404 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612434 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.612730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.612582 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k996m\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.651826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.624266 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.651826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.650163 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.651826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.650462 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.652102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.651980 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.664690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.653520 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.664690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.656895 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:12:20.688281 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.688204 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.690086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.689781 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.769773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.755468 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k996m\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.837208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.837089 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.839871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.839839 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\") pod \"e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:20.847718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.845241 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:12:20.847718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.845306 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:20.943725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:20.942823 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:21.198723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.194229 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:12:21.198723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.194313 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:12:21.330363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.330327 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c2745f-5a87-4d9d-b92e-f07c71dddc86" path="/var/lib/kubelet/pods/e9c2745f-5a87-4d9d-b92e-f07c71dddc86/volumes" Oct 30 20:12:21.373726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.373437 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.622089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.621981 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.803192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.803150 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:12:21.838834 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:12:21.825789 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476fa658_bc72_41c9_8ad7_b718b4f3280f.slice/crio-472972b62aaef0df8af76fdc6f01141d5685d77a217322851d49487115281c1f WatchSource:0}: Error finding container 472972b62aaef0df8af76fdc6f01141d5685d77a217322851d49487115281c1f: Status 404 returned error can't find the container with id 472972b62aaef0df8af76fdc6f01141d5685d77a217322851d49487115281c1f Oct 30 20:12:21.838834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.838634 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:21.838834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.838739 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:21.838834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:21.838784 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:22.243377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.243333 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.321445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.321405 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerStarted","Data":"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af"} Oct 30 20:12:22.321641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.321473 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerStarted","Data":"472972b62aaef0df8af76fdc6f01141d5685d77a217322851d49487115281c1f"} Oct 30 20:12:22.486845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.486810 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.497319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.497230 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podStartSLOduration=2.497216639 podStartE2EDuration="2.497216639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:22.445352372 +0000 UTC m=+1837.289024038" watchObservedRunningTime="2026-10-30 20:12:22.497216639 +0000 UTC m=+1837.340888268" Oct 30 20:12:22.923832 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.923784 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:22.925463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.925437 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:22.926290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.926268 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:12:22.947740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.947711 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:22.953584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:22.953555 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:23.332082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:23.332009 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:23.332749 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:23.332727 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:12:25.305042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.304997 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.398814 4524 generic.go:356] "Generic (PLEG): container finished" podID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerID="fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09" exitCode=0 Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.400018 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerDied","Data":"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09"} Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.400527 4524 scope.go:122] "RemoveContainer" containerID="fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09" Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.403015 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.403099 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:25.408722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.403148 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:25.837772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:25.832573 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.119769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:26.115481 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.857722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:26.853237 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerStarted","Data":"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d"} Oct 30 20:12:26.941806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:26.933571 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:12:26.941806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:26.933844 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:26.941806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:26.933934 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:27.456722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:27.449413 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.782768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:27.780717 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.895072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:27.895037 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:27.895649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:27.895618 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:28.057710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.057559 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.061842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.061789 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:12:28.062302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.062271 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" containerID="cri-o://566a0437c1d0a766c6e5f872e0f1819d76150572922010f18fc8a516d4eaddec" gracePeriod=30 Oct 30 20:12:28.318876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.318689 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.890915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.890881 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:28.891091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:28.890939 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:29.890521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:29.890414 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:29.890521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:29.890474 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:29.984958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:29.984912 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.723728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:30.722338 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.909723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:30.897687 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:30.909723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:30.897768 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:31.562952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:31.562911 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.831952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:31.818457 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.895152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:31.895111 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:31.895567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:31.895533 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:32.891626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:32.891594 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:32.892057 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:32.891651 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:33.326919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.326883 4524 generic.go:356] "Generic (PLEG): container finished" podID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerID="566a0437c1d0a766c6e5f872e0f1819d76150572922010f18fc8a516d4eaddec" exitCode=0 Oct 30 20:12:33.327074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.326962 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerDied","Data":"566a0437c1d0a766c6e5f872e0f1819d76150572922010f18fc8a516d4eaddec"} Oct 30 20:12:33.327074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.327045 4524 scope.go:122] "RemoveContainer" containerID="21ace7a70aee672c1584e88f359c7221e3cf86e7f177511de28182a943b8d351" Oct 30 20:12:33.431184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.431158 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:12:33.619640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.619283 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.619640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.619480 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.619940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.619913 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.619959 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.619985 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.620133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620078 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620137 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620176 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620181 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.620350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620219 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620257 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:33.620350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620314 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\") pod \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\" (UID: \"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6\") " Oct 30 20:12:33.620503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.620356 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.621198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.621168 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.621321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.621225 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.621321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.621252 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.621321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.621267 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.621321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.621281 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.623235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.623209 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.623941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.623917 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:33.624135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.624081 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "kube-api-access-ldxg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:33.626159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.626122 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo" pod "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" (UID: "ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:33.722513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.722478 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.722513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.722510 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.722513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.722519 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.722763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.722527 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.722763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.722538 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxg7\" (UniqueName: \"kubernetes.io/projected/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6-kube-api-access-ldxg7\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.903773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.896128 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:33.903773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.896183 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:33.903773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.899076 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:12:33.957610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.955361 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:33.957610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.955399 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:33.957610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.955596 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:33.957610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:33.955621 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:34.110843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.092245 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:12:34.110843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.092384 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.219802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.194236 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7961ed-serving-cert\"" Oct 30 20:12:34.224027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.223989 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.233129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.233093 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btxb\" (UniqueName: \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.233303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.233281 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.233384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.233345 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.233460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.233433 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.352786 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.352734 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" event={"ID":"ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6","Type":"ContainerDied","Data":"5d115d42c628399d4456fdb6a9910e3a9bc34968b5167d94506dd12c39634276"} Oct 30 20:12:34.353076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.353054 4524 scope.go:122] "RemoveContainer" containerID="566a0437c1d0a766c6e5f872e0f1819d76150572922010f18fc8a516d4eaddec" Oct 30 20:12:34.353385 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.353367 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4" Oct 30 20:12:34.543999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.543908 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btxb\" (UniqueName: \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.544451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.544425 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.544640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.544623 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.544865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.544848 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.547057 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:12:34.547036 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-serving-cert: secret "maas-api-e2e-shared-a-w3-7961ed-serving-cert" not found Oct 30 20:12:34.547284 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:12:34.547269 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls podName:7e314883-d870-4fa3-9396-909f32ddae37 nodeName:}" failed. No retries permitted until 2026-10-30 20:12:35.047244877 +0000 UTC m=+1849.890916498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls") pod "maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" (UID: "7e314883-d870-4fa3-9396-909f32ddae37") : secret "maas-api-e2e-shared-a-w3-7961ed-serving-cert" not found Oct 30 20:12:34.548394 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.548371 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.608318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.595186 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.632110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.632062 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.692279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.692245 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btxb\" (UniqueName: \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:34.796586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.796500 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:12:34.808529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.808496 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:12:34.814528 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.814497 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7ca5342-openshift-default-9fd5774d7-jrnd4"] Oct 30 20:12:34.858586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.858548 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.890563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.890527 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:12:34.890852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:34.890820 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 20:12:35.077809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.077721 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:35.080388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.080360 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:35.134748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.134715 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:35.201212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.201091 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.307710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.307668 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" path="/var/lib/kubelet/pods/ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6/volumes" Oct 30 20:12:35.530741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.530694 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:12:35.531055 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:12:35.531023 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e314883_d870_4fa3_9396_909f32ddae37.slice/crio-8cd1f6407e3762ed9dd3d364555f67b0f95b68ad1ecf160a92ac36382b5bb5c6 WatchSource:0}: Error finding container 8cd1f6407e3762ed9dd3d364555f67b0f95b68ad1ecf160a92ac36382b5bb5c6: Status 404 returned error can't find the container with id 8cd1f6407e3762ed9dd3d364555f67b0f95b68ad1ecf160a92ac36382b5bb5c6 Oct 30 20:12:35.895423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.895396 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:12:35.896626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:35.896606 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:12:36.362035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:36.361942 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" event={"ID":"7e314883-d870-4fa3-9396-909f32ddae37","Type":"ContainerStarted","Data":"8cd1f6407e3762ed9dd3d364555f67b0f95b68ad1ecf160a92ac36382b5bb5c6"} Oct 30 20:12:37.379297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:37.379231 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" event={"ID":"7e314883-d870-4fa3-9396-909f32ddae37","Type":"ContainerStarted","Data":"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad"} Oct 30 20:12:37.379794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:37.379446 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:37.500401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:37.500355 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" podStartSLOduration=3.471874991 podStartE2EDuration="4.500340856s" totalImagesPullingTime="1.028465865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:37.497033792 +0000 UTC m=+1852.340705437" watchObservedRunningTime="2026-10-30 20:12:37.500340856 +0000 UTC m=+1852.344012485" Oct 30 20:12:39.441120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.441075 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.890108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.751044 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.895752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.892994 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:12:39.895752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.894268 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:39.895752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.894292 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5d3de7-b4dc-4df5-8f8e-9f6109bc42f6" containerName="istio-proxy" Oct 30 20:12:39.903255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:39.902191 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.035660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.010071 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-37732de7\"" Oct 30 20:12:40.035660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.010251 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-37732de7-dockercfg-b8hm2\"" Oct 30 20:12:40.035660 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.028800 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.056181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.056142 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.056381 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.056200 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zp98\" (UniqueName: \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.214774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.166070 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.314894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.314851 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zp98\" (UniqueName: \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.316320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.174718 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.397726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.393494 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.433806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.419759 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:12:40.443478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.443418 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zp98\" (UniqueName: \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:40.568770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:40.559988 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:41.202800 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:12:41.201768 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8869a5_210e_4527_8c8c_85e72233665e.slice/crio-9929b42b86231cdcfe774fb4811810f2b52360ec3020c18092054a0eb4999328 WatchSource:0}: Error finding container 9929b42b86231cdcfe774fb4811810f2b52360ec3020c18092054a0eb4999328: Status 404 returned error can't find the container with id 9929b42b86231cdcfe774fb4811810f2b52360ec3020c18092054a0eb4999328 Oct 30 20:12:41.203017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:41.202926 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:12:41.457485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:41.457401 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" event={"ID":"8f8869a5-210e-4527-8c8c-85e72233665e","Type":"ContainerStarted","Data":"ea701f6e43d9cef5ca1059f79178ebef2770377c6cf18b4d1e3c9c83d6771a90"} Oct 30 20:12:41.457485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:41.457456 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" event={"ID":"8f8869a5-210e-4527-8c8c-85e72233665e","Type":"ContainerStarted","Data":"9929b42b86231cdcfe774fb4811810f2b52360ec3020c18092054a0eb4999328"} Oct 30 20:12:41.458035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:41.457506 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:41.577476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:41.577419 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" podStartSLOduration=2.577405676 podStartE2EDuration="2.577405676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:41.574166836 +0000 UTC m=+1856.417838489" watchObservedRunningTime="2026-10-30 20:12:41.577405676 +0000 UTC m=+1856.421077305" Oct 30 20:12:42.788071 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:12:42.788030 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:12:43.391874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:43.391843 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:12:44.872013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:44.871971 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.799724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:45.797824 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:46.720489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:46.720458 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:50.104211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:50.104177 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:50.702722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:50.695619 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.104442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.104359 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:12:51.111343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.111316 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.170098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.170065 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:12:51.172522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.172494 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f3736ef9-openshift-default-dockercfg-tchkp\"" Oct 30 20:12:51.216716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.216663 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.216887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.216738 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.216962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.216933 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217045 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t97t\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217141 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217266 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217319 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217419 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.217534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.217517 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.240001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.239961 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.319156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319128 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319184 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t97t\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319233 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319294 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319318 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319396 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319489 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319581 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319617 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319754 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319790 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.319946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319922 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.320079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.319959 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.320079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.320030 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.321992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.321970 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.322063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.322047 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.381746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.381679 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.381902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.381881 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t97t\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\") pod \"e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.459356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.459071 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:51.482649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.482450 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.947994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:51.947919 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.071675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.071643 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:12:52.109563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.109529 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:52.109897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.109597 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:52.109897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.109627 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:52.202328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.202249 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.470257 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.470173 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:12:52.607020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.606976 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerStarted","Data":"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f"} Oct 30 20:12:52.607197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.607043 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerStarted","Data":"4fb96801e31446d957e4098a16df62d25a4b9607105d74b689f37d1acf542548"} Oct 30 20:12:52.732321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:52.732209 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podStartSLOduration=1.7321893560000001 podStartE2EDuration="1.732189356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:52.729003623 +0000 UTC m=+1867.572675254" watchObservedRunningTime="2026-10-30 20:12:52.732189356 +0000 UTC m=+1867.575860986" Oct 30 20:12:53.036384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.035161 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.459941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.459905 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:53.467990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.467953 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:53.613750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.613715 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:53.614691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.614649 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:12:53.799310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:53.799224 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:54.474904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:54.474815 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:55.775910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:55.775874 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.016237 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:56.016204 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.268238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:56.268192 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.544139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:56.544033 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.084914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:59.084875 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.317654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:59.317615 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.856429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:12:59.856292 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.127533 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.127443 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.307994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.307961 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:13:00.839434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.839380 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:13:00.863941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.851719 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:00.904157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.902515 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2fs\" (UniqueName: \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:00.904835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.904811 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:00.968579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.968547 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3736ef9-dockercfg-smdmw\"" Oct 30 20:13:00.968579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:00.968556 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3736ef9\"" Oct 30 20:13:01.006900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.006868 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2fs\" (UniqueName: \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.007042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.007008 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.007682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.007662 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.071646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.071612 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2fs\" (UniqueName: \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\") pod \"payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.218730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.218671 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.605111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.605017 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:13:01.610336 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:13:01.610305 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a81e100_93ff_40e6_9d59_6073deb36750.slice/crio-fc47eb5b14946b0de3fd85e5c81ca68f0c087da01d3a74888cb0fcbc9f6bfbc6 WatchSource:0}: Error finding container fc47eb5b14946b0de3fd85e5c81ca68f0c087da01d3a74888cb0fcbc9f6bfbc6: Status 404 returned error can't find the container with id fc47eb5b14946b0de3fd85e5c81ca68f0c087da01d3a74888cb0fcbc9f6bfbc6 Oct 30 20:13:01.838505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.838457 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" event={"ID":"4a81e100-93ff-40e6-9d59-6073deb36750","Type":"ContainerStarted","Data":"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77"} Oct 30 20:13:01.838693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.838528 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:01.838693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.838545 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" event={"ID":"4a81e100-93ff-40e6-9d59-6073deb36750","Type":"ContainerStarted","Data":"fc47eb5b14946b0de3fd85e5c81ca68f0c087da01d3a74888cb0fcbc9f6bfbc6"} Oct 30 20:13:01.958348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:01.958298 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" podStartSLOduration=1.958282712 podStartE2EDuration="1.958282712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:01.955489683 +0000 UTC m=+1876.799161313" watchObservedRunningTime="2026-10-30 20:13:01.958282712 +0000 UTC m=+1876.801954340" Oct 30 20:13:02.138147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:02.138108 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:06.876380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:06.876341 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.185726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:07.178950 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.725111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:07.723414 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:13:07.731525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:07.731487 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:07.742746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:07.742711 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:13:08.008534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.004319 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f44ec2-dockercfg-87gjr\"" Oct 30 20:13:08.008534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.004418 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f44ec2\"" Oct 30 20:13:08.031500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.031468 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.031842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.031806 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnrz\" (UniqueName: \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.133854 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.133814 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnrz\" (UniqueName: \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.134046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.133911 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.134619 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.134591 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.195546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.195507 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnrz\" (UniqueName: \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\") pod \"payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.337795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.337687 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:08.725873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:08.725842 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:13:08.726426 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:13:08.726403 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb656d072_a2a0_4021_b863_0d5e2ec2b3b5.slice/crio-b2e1d72e8722de207abf6e2b804d078afa334fed27df8821b19f9a77b51b511b WatchSource:0}: Error finding container b2e1d72e8722de207abf6e2b804d078afa334fed27df8821b19f9a77b51b511b: Status 404 returned error can't find the container with id b2e1d72e8722de207abf6e2b804d078afa334fed27df8821b19f9a77b51b511b Oct 30 20:13:09.048795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:09.048577 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" event={"ID":"b656d072-a2a0-4021-b863-0d5e2ec2b3b5","Type":"ContainerStarted","Data":"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4"} Oct 30 20:13:09.048795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:09.048640 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:09.048795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:09.048658 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" event={"ID":"b656d072-a2a0-4021-b863-0d5e2ec2b3b5","Type":"ContainerStarted","Data":"b2e1d72e8722de207abf6e2b804d078afa334fed27df8821b19f9a77b51b511b"} Oct 30 20:13:09.167628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:09.167578 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" podStartSLOduration=2.167564683 podStartE2EDuration="2.167564683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:09.164842669 +0000 UTC m=+1884.008514299" watchObservedRunningTime="2026-10-30 20:13:09.167564683 +0000 UTC m=+1884.011236312" Oct 30 20:13:12.523723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:12.523409 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:12.786944 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:12.786865 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:13:12.848686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:12.848658 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:13:17.471729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:17.471052 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.058588 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:20.058559 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:13:22.251319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:22.251237 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.475762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:22.475483 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.689298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:22.689266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.403795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:27.402177 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.652762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:27.652471 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:28.206907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:28.206867 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.504865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:32.504804 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:34.540819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.540785 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:13:34.541298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.541029 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" containerID="cri-o://7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428" gracePeriod=30 Oct 30 20:13:34.591043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.591014 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:13:34.591781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.591688 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" containerID="cri-o://65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede" gracePeriod=30 Oct 30 20:13:34.600054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.600023 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:13:34.600377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:34.600333 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" containerID="cri-o://0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9" gracePeriod=30 Oct 30 20:13:36.997877 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:36.997843 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:13:37.004007 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.003975 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.057303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.057275 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-37732de7-serving-cert\"" Oct 30 20:13:37.084154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.084122 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.084336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.084285 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.084336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.084315 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.084470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.084425 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v7v\" (UniqueName: \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.114689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.114663 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:13:37.186028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.185986 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.186222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.186137 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.186222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.186170 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.186358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.186268 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v7v\" (UniqueName: \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.186358 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:37.186345 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-serving-cert: secret "maas-api-e2e-mt-37732de7-serving-cert" not found Oct 30 20:13:37.186468 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:37.186431 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls podName:49ea0538-d11e-4806-a36e-0a1cb1253225 nodeName:}" failed. No retries permitted until 2026-10-30 20:13:37.686411032 +0000 UTC m=+1912.530082656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls") pod "maas-api-e2e-mt-37732de7-7b44f7679-66fpt" (UID: "49ea0538-d11e-4806-a36e-0a1cb1253225") : secret "maas-api-e2e-mt-37732de7-serving-cert" not found Oct 30 20:13:37.186868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.186846 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.188905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.188879 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.251346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.251278 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v7v\" (UniqueName: \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.315118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.315088 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.573779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.564663 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.764457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.764414 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.767064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.767037 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-66fpt\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:37.918406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:37.918361 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:38.305303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:38.305272 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:13:38.305937 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:13:38.305906 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea0538_d11e_4806_a36e_0a1cb1253225.slice/crio-afcf7fc5144451d4a9a57e5a819d868d354556632f62388c98e8245d1a082655 WatchSource:0}: Error finding container afcf7fc5144451d4a9a57e5a819d868d354556632f62388c98e8245d1a082655: Status 404 returned error can't find the container with id afcf7fc5144451d4a9a57e5a819d868d354556632f62388c98e8245d1a082655 Oct 30 20:13:38.390424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:38.390383 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" event={"ID":"49ea0538-d11e-4806-a36e-0a1cb1253225","Type":"ContainerStarted","Data":"afcf7fc5144451d4a9a57e5a819d868d354556632f62388c98e8245d1a082655"} Oct 30 20:13:38.615934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:38.615847 4524 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:13:38.615934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:38.615914 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:13:39.401027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:39.400989 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" event={"ID":"49ea0538-d11e-4806-a36e-0a1cb1253225","Type":"ContainerStarted","Data":"8aee1b6a0c994f8b5ca0062cd4d804c19cb2cf892acd1746e3d35a882540ebe0"} Oct 30 20:13:39.401454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:39.401056 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:39.549022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:39.548908 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" podStartSLOduration=2.553815421 podStartE2EDuration="3.548891061s" totalImagesPullingTime="995.07564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:39.545484278 +0000 UTC m=+1914.389155919" watchObservedRunningTime="2026-10-30 20:13:39.548891061 +0000 UTC m=+1914.392562693" Oct 30 20:13:40.060021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.059988 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:13:40.124635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.124604 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:13:40.128779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.128755 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:13:40.170754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170688 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.170907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170809 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.170907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170857 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.171027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170894 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170959 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170987 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.171027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.170990 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.171027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171020 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171093 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171122 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171182 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171217 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171223 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.171286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171283 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171325 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171359 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171414 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171450 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171496 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171546 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.171595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171591 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171732 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171780 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\") pod \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\" (UID: \"73c96e0c-4fc8-422f-8ca0-e366fa90e63e\") " Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171824 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171852 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171888 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171891 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.172008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.171966 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172049 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172091 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172129 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172167 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172174 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\") pod \"dfbf3037-13bd-4455-b563-f2c43171e660\" (UID: \"dfbf3037-13bd-4455-b563-f2c43171e660\") " Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172263 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\") pod \"8529de96-8528-4d08-add2-5e5644eaa580\" (UID: \"8529de96-8528-4d08-add2-5e5644eaa580\") " Oct 30 20:13:40.172393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172357 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.172779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.172531 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:40.173342 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173095 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.173444 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173388 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173444 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173411 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173444 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173426 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173444 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173439 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173453 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173468 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173483 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173496 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173510 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8529de96-8528-4d08-add2-5e5644eaa580-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.173656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173524 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf3037-13bd-4455-b563-f2c43171e660-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.174395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173766 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.174395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.173951 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.174395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.174091 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.174395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.174291 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.174395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.174376 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.175284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.175258 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "kube-api-access-l78vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.176101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.176068 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "kube-api-access-2trkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.177144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177117 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:40.177343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177318 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.177467 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177373 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.177534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177486 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.177622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177602 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:40.177622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177615 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy" pod "8529de96-8528-4d08-add2-5e5644eaa580" (UID: "8529de96-8528-4d08-add2-5e5644eaa580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.177776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177716 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:40.177834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.177792 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token" pod "dfbf3037-13bd-4455-b563-f2c43171e660" (UID: "dfbf3037-13bd-4455-b563-f2c43171e660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.178249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.178221 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:40.179034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.179016 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv" pod "73c96e0c-4fc8-422f-8ca0-e366fa90e63e" (UID: "73c96e0c-4fc8-422f-8ca0-e366fa90e63e"). InnerVolumeSpecName "kube-api-access-6vkfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:40.275919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275879 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.275919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275916 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275934 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275949 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275964 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275978 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf3037-13bd-4455-b563-f2c43171e660-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.275992 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276005 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276019 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trkc\" (UniqueName: \"kubernetes.io/projected/dfbf3037-13bd-4455-b563-f2c43171e660-kube-api-access-2trkc\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276036 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8529de96-8528-4d08-add2-5e5644eaa580-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276049 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8529de96-8528-4d08-add2-5e5644eaa580-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276063 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276077 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkfv\" (UniqueName: \"kubernetes.io/projected/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-kube-api-access-6vkfv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276092 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78vl\" (UniqueName: \"kubernetes.io/projected/8529de96-8528-4d08-add2-5e5644eaa580-kube-api-access-l78vl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276107 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276123 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73c96e0c-4fc8-422f-8ca0-e366fa90e63e-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.276501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.276137 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf3037-13bd-4455-b563-f2c43171e660-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:40.410291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.410261 4524 generic.go:356] "Generic (PLEG): container finished" podID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerID="7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428" exitCode=0 Oct 30 20:13:40.410770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.410312 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" Oct 30 20:13:40.410770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.410338 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" event={"ID":"73c96e0c-4fc8-422f-8ca0-e366fa90e63e","Type":"ContainerDied","Data":"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428"} Oct 30 20:13:40.410770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.410392 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt" event={"ID":"73c96e0c-4fc8-422f-8ca0-e366fa90e63e","Type":"ContainerDied","Data":"f22eaf32685217aee2294ccce3fd9f7100523ba3dcb8332678a54a1c303c9b77"} Oct 30 20:13:40.410770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.410425 4524 scope.go:122] "RemoveContainer" containerID="7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428" Oct 30 20:13:40.412209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.412190 4524 generic.go:356] "Generic (PLEG): container finished" podID="8529de96-8528-4d08-add2-5e5644eaa580" containerID="65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede" exitCode=0 Oct 30 20:13:40.412315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.412299 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" event={"ID":"8529de96-8528-4d08-add2-5e5644eaa580","Type":"ContainerDied","Data":"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede"} Oct 30 20:13:40.412387 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.412344 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" event={"ID":"8529de96-8528-4d08-add2-5e5644eaa580","Type":"ContainerDied","Data":"baf70453a34517041f632df78ec465f8a655eeb34fa0e4f9c0623c293ab7c47c"} Oct 30 20:13:40.412441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.412430 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl" Oct 30 20:13:40.416968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.416919 4524 generic.go:356] "Generic (PLEG): container finished" podID="dfbf3037-13bd-4455-b563-f2c43171e660" containerID="0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9" exitCode=0 Oct 30 20:13:40.443369 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.437366 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" event={"ID":"dfbf3037-13bd-4455-b563-f2c43171e660","Type":"ContainerDied","Data":"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9"} Oct 30 20:13:40.443369 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.437438 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" event={"ID":"dfbf3037-13bd-4455-b563-f2c43171e660","Type":"ContainerDied","Data":"1783aff7dfaa9ab52a679e68e6c589afc36478aaf120f6d107d9c25065f79db4"} Oct 30 20:13:40.455511 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.455418 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd" Oct 30 20:13:40.463890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.463868 4524 scope.go:122] "RemoveContainer" containerID="7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428" Oct 30 20:13:40.464356 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:40.464293 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428\": container with ID starting with 7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428 not found: ID does not exist" containerID="7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428" Oct 30 20:13:40.464356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.464334 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428"} err="failed to get container status \"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428\": rpc error: code = NotFound desc = could not find container \"7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428\": container with ID starting with 7eafc5a6cc5ed0fae1191c6dad988536d7355c9df0d0b7941465f98874bb1428 not found: ID does not exist" Oct 30 20:13:40.464809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.464359 4524 scope.go:122] "RemoveContainer" containerID="65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede" Oct 30 20:13:40.479267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.479249 4524 scope.go:122] "RemoveContainer" containerID="65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede" Oct 30 20:13:40.479541 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:40.479521 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede\": container with ID starting with 65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede not found: ID does not exist" containerID="65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede" Oct 30 20:13:40.479606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.479549 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede"} err="failed to get container status \"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede\": rpc error: code = NotFound desc = could not find container \"65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede\": container with ID starting with 65b2058d90b5b632f6d94687b4bee1380bda387277c01822364370ca4e9d1ede not found: ID does not exist" Oct 30 20:13:40.479606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.479566 4524 scope.go:122] "RemoveContainer" containerID="0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9" Oct 30 20:13:40.492448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.492430 4524 scope.go:122] "RemoveContainer" containerID="0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9" Oct 30 20:13:40.492748 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:40.492679 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9\": container with ID starting with 0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9 not found: ID does not exist" containerID="0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9" Oct 30 20:13:40.492856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.492756 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9"} err="failed to get container status \"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9\": rpc error: code = NotFound desc = could not find container \"0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9\": container with ID starting with 0aa026ac935b4ffe221831b71f36660ee4e8fa84d36ce2dcb0a969b263b649a9 not found: ID does not exist" Oct 30 20:13:40.671231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.671115 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:13:40.677754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.677724 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-pc4pd"] Oct 30 20:13:40.822858 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.816994 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:13:40.822858 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.822746 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-xlzbl"] Oct 30 20:13:40.899876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.899841 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:13:40.913378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:40.913338 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5vhjt"] Oct 30 20:13:41.302091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:41.302060 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" path="/var/lib/kubelet/pods/73c96e0c-4fc8-422f-8ca0-e366fa90e63e/volumes" Oct 30 20:13:41.302609 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:41.302594 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8529de96-8528-4d08-add2-5e5644eaa580" path="/var/lib/kubelet/pods/8529de96-8528-4d08-add2-5e5644eaa580/volumes" Oct 30 20:13:41.303125 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:41.303111 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" path="/var/lib/kubelet/pods/dfbf3037-13bd-4455-b563-f2c43171e660/volumes" Oct 30 20:13:42.592730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:42.591837 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:42.791364 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:42.791321 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:13:45.176445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:45.176409 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.436659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:45.436582 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:13:47.925743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:47.922854 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:50.504234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.504196 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:13:50.504688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.504407 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" containerID="cri-o://77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086" gracePeriod=30 Oct 30 20:13:50.568726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.558854 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:13:50.568726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.559088 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" containerID="cri-o://2a60d3cc3c25f55c65f1efc1d330c3c19d28257b96fa759826ab384dcd43a3e2" gracePeriod=30 Oct 30 20:13:50.589723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.575427 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:13:50.589723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:50.575679 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" containerID="cri-o://e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d" gracePeriod=30 Oct 30 20:13:51.899723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:51.896470 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.401450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.401417 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.924720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.924085 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:13:52.930670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.930641 4524 cpu_manager.go:414] "containerMap: removing container" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" Oct 30 20:13:52.931632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.930916 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" Oct 30 20:13:52.931866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.931852 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" Oct 30 20:13:52.931979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.931965 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" Oct 30 20:13:52.932109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.932096 4524 cpu_manager.go:414] "containerMap: removing container" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" Oct 30 20:13:52.932210 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.932199 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" Oct 30 20:13:52.932477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.932463 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8529de96-8528-4d08-add2-5e5644eaa580" containerName="istio-proxy" Oct 30 20:13:52.932585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.932573 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c96e0c-4fc8-422f-8ca0-e366fa90e63e" containerName="istio-proxy" Oct 30 20:13:52.932710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.932687 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbf3037-13bd-4455-b563-f2c43171e660" containerName="istio-proxy" Oct 30 20:13:52.948826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:52.948646 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.009859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.005831 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bb565e-openshift-default-dockercfg-5dh9k\"" Oct 30 20:13:53.051799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.050018 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.066751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.052073 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.067171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.067140 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.067534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.067501 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.118728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.106215 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.145070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.145030 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvvn\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.145435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.145399 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.145728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.145676 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.146043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.146023 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.424111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.168512 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.560040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.546942 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.566365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.566057 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvvn\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.566365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.566175 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.566365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.566317 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.573969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.573818 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.574408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.574205 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.574408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.574313 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.595052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.595019 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.596169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.596140 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.596658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.596634 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.641384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.574638 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.641993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.641961 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.689750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.689655 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.787724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.766214 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.831223 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.831183 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.832692 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.832653 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.934499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.934448 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:13:53.990659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.990583 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:53.994140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:53.994102 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:54.024727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.020309 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvvn\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\") pod \"e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:54.099947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.099913 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:13:54.221416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.221099 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:54.817913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.811027 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:13:54.821934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.821888 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:13:54.822141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.822043 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:13:54.822381 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.822337 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:13:54.996333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:54.980834 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:55.238206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:55.235448 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerStarted","Data":"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a"} Oct 30 20:13:55.238206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:55.235513 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerStarted","Data":"775dc62d9793f23b62425806dee7d63243e7306ac56f18ddb93a94805a3500e6"} Oct 30 20:13:55.391666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:55.391595 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podStartSLOduration=3.391573452 podStartE2EDuration="3.391573452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:55.362660959 +0000 UTC m=+1930.206332625" watchObservedRunningTime="2026-10-30 20:13:55.391573452 +0000 UTC m=+1930.235245085" Oct 30 20:13:56.199233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.198884 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:13:56.204948 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.204924 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:13:56.222742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.222238 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:56.228168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.228143 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:56.245115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.244937 4524 generic.go:356] "Generic (PLEG): container finished" podID="25d64291-d039-4acc-9240-6182267c1848" containerID="2a60d3cc3c25f55c65f1efc1d330c3c19d28257b96fa759826ab384dcd43a3e2" exitCode=0 Oct 30 20:13:56.245115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.245094 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" event={"ID":"25d64291-d039-4acc-9240-6182267c1848","Type":"ContainerDied","Data":"2a60d3cc3c25f55c65f1efc1d330c3c19d28257b96fa759826ab384dcd43a3e2"} Oct 30 20:13:56.292747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.289162 4524 generic.go:356] "Generic (PLEG): container finished" podID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerID="e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d" exitCode=0 Oct 30 20:13:56.292747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.289224 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" Oct 30 20:13:56.292747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.289280 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" event={"ID":"47da3bba-58fa-47a1-b7b5-cc9d2462c22c","Type":"ContainerDied","Data":"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d"} Oct 30 20:13:56.292747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.289328 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5" event={"ID":"47da3bba-58fa-47a1-b7b5-cc9d2462c22c","Type":"ContainerDied","Data":"cad14077215af53d73348a3898704228a8bad66d3d4d0fb24f60ebdb97381b4c"} Oct 30 20:13:56.292747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.289359 4524 scope.go:122] "RemoveContainer" containerID="e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d" Oct 30 20:13:56.293274 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.292978 4524 generic.go:356] "Generic (PLEG): container finished" podID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerID="77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086" exitCode=0 Oct 30 20:13:56.294186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.294165 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" Oct 30 20:13:56.294325 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.294295 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" event={"ID":"b6cb227a-c86c-4838-80ef-756d2af0ff40","Type":"ContainerDied","Data":"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086"} Oct 30 20:13:56.294391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.294339 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:56.294391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.294366 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh" event={"ID":"b6cb227a-c86c-4838-80ef-756d2af0ff40","Type":"ContainerDied","Data":"d102b0d56625cd619d63932381c52da13fbba0b4738e9b8874779a4cab172881"} Oct 30 20:13:56.295829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.295811 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:13:56.303799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.303784 4524 scope.go:122] "RemoveContainer" containerID="e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d" Oct 30 20:13:56.304064 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:56.304038 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d\": container with ID starting with e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d not found: ID does not exist" containerID="e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d" Oct 30 20:13:56.304131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.304077 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d"} err="failed to get container status \"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d\": rpc error: code = NotFound desc = could not find container \"e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d\": container with ID starting with e193931924652e8358ea243c9851368d2525bf9eff0a184adbba23816d32d56d not found: ID does not exist" Oct 30 20:13:56.304131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.304099 4524 scope.go:122] "RemoveContainer" containerID="77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086" Oct 30 20:13:56.304131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.304116 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:13:56.314580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.314531 4524 scope.go:122] "RemoveContainer" containerID="77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086" Oct 30 20:13:56.314877 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:13:56.314857 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086\": container with ID starting with 77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086 not found: ID does not exist" containerID="77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086" Oct 30 20:13:56.314970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.314886 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086"} err="failed to get container status \"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086\": rpc error: code = NotFound desc = could not find container \"77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086\": container with ID starting with 77458787c26a71571155311f9f9023bcb0fd76956e6b8138e3f47e973ded2086 not found: ID does not exist" Oct 30 20:13:56.336427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336401 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.336532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336480 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336545 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336586 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336657 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336685 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.336780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336754 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336802 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.336928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336845 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.336928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336903 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336953 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336984 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.336989 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337043 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337042 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337067 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.337080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337079 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337119 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337173 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337247 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337301 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337386 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.337417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337393 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\") pod \"b6cb227a-c86c-4838-80ef-756d2af0ff40\" (UID: \"b6cb227a-c86c-4838-80ef-756d2af0ff40\") " Oct 30 20:13:56.337774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337439 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337467 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\") pod \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\" (UID: \"47da3bba-58fa-47a1-b7b5-cc9d2462c22c\") " Oct 30 20:13:56.337943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.337919 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.338081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338053 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:56.338263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338241 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338266 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cb227a-c86c-4838-80ef-756d2af0ff40-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338283 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338298 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338312 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338326 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338341 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338357 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.338606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.338371 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.340207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.340180 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.340207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.340180 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "kube-api-access-cz4dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.340404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.340389 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.340764 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.340727 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.340867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.340788 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.341273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.341249 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.341273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.341256 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:56.341457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.341433 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg" pod "b6cb227a-c86c-4838-80ef-756d2af0ff40" (UID: "b6cb227a-c86c-4838-80ef-756d2af0ff40"). InnerVolumeSpecName "kube-api-access-zx8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.341504 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.341474 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo" pod "47da3bba-58fa-47a1-b7b5-cc9d2462c22c" (UID: "47da3bba-58fa-47a1-b7b5-cc9d2462c22c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:56.439478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439396 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.439478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439462 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.439727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439498 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.439797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439692 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.439797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439736 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.439908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439817 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.439908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439861 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.440014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.439941 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.440070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440015 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.440070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440039 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:56.440174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440151 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.440228 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440183 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\") pod \"25d64291-d039-4acc-9240-6182267c1848\" (UID: \"25d64291-d039-4acc-9240-6182267c1848\") " Oct 30 20:13:56.440601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440562 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.440601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.440573 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.441096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441077 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441101 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441118 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441130 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441143 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441156 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441171 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cb227a-c86c-4838-80ef-756d2af0ff40-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441184 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441197 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d64291-d039-4acc-9240-6182267c1848-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441212 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441226 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441243 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8xg\" (UniqueName: \"kubernetes.io/projected/b6cb227a-c86c-4838-80ef-756d2af0ff40-kube-api-access-zx8xg\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441259 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.441604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.441273 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz4dv\" (UniqueName: \"kubernetes.io/projected/47da3bba-58fa-47a1-b7b5-cc9d2462c22c-kube-api-access-cz4dv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.442960 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.442923 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:56.443667 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.443641 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.443957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.443931 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "kube-api-access-zs9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:56.444447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.444421 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo" pod "25d64291-d039-4acc-9240-6182267c1848" (UID: "25d64291-d039-4acc-9240-6182267c1848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:56.542950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.542868 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9rq\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-kube-api-access-zs9rq\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.542950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.542905 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d64291-d039-4acc-9240-6182267c1848-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.542950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.542922 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d64291-d039-4acc-9240-6182267c1848-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:56.542950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:56.542936 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d64291-d039-4acc-9240-6182267c1848-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:13:57.061506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.061456 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:13:57.067669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.067640 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-6r9c5"] Oct 30 20:13:57.169291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.169203 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:13:57.175495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.175463 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fhpsh"] Oct 30 20:13:57.303688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.303659 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" Oct 30 20:13:57.305604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.305576 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" path="/var/lib/kubelet/pods/47da3bba-58fa-47a1-b7b5-cc9d2462c22c/volumes" Oct 30 20:13:57.306289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.306265 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" path="/var/lib/kubelet/pods/b6cb227a-c86c-4838-80ef-756d2af0ff40/volumes" Oct 30 20:13:57.311741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.306854 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln" event={"ID":"25d64291-d039-4acc-9240-6182267c1848","Type":"ContainerDied","Data":"8774f70011db1553b7002af7c307f60a4efee0f27896b16d3ef272e480d8caf4"} Oct 30 20:13:57.311741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.306909 4524 scope.go:122] "RemoveContainer" containerID="2a60d3cc3c25f55c65f1efc1d330c3c19d28257b96fa759826ab384dcd43a3e2" Oct 30 20:13:57.476839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.476749 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:13:57.480206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:57.480181 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-d7kln"] Oct 30 20:13:58.647969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:58.647931 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:58.914950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:58.914871 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.217874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:59.210719 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.320914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:13:59.320872 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d64291-d039-4acc-9240-6182267c1848" path="/var/lib/kubelet/pods/25d64291-d039-4acc-9240-6182267c1848/volumes" Oct 30 20:14:00.332330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.332296 4524 generic.go:356] "Generic (PLEG): container finished" podID="92e6774b-46b8-49da-af50-2706f3717e27" containerID="3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c" exitCode=0 Oct 30 20:14:00.332797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.332368 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerDied","Data":"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c"} Oct 30 20:14:00.332896 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.332871 4524 scope.go:122] "RemoveContainer" containerID="3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c" Oct 30 20:14:00.334773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.334746 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:00.334851 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.334817 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:00.334899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:00.334852 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:01.341198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.341164 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerStarted","Data":"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0"} Oct 30 20:14:01.343004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.342979 4524 generic.go:356] "Generic (PLEG): container finished" podID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerID="ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af" exitCode=0 Oct 30 20:14:01.343146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.343038 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerDied","Data":"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af"} Oct 30 20:14:01.343463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.343445 4524 scope.go:122] "RemoveContainer" containerID="ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af" Oct 30 20:14:01.345078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.345049 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:01.345141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.345115 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:01.345178 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:01.345147 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:02.079168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:02.078013 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.079168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:02.078155 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:14:02.080192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:02.080167 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:02.080343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:02.080214 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:02.633726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:02.626666 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:03.059304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.030562 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerStarted","Data":"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a"} Oct 30 20:14:03.204764 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.155591 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:03.205081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.205041 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:03.205235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.180107 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.207130 4524 cpu_manager.go:414] "containerMap: removing container" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.244751 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.244860 4524 cpu_manager.go:414] "containerMap: removing container" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.244870 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.244928 4524 cpu_manager.go:414] "containerMap: removing container" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.244937 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.245442 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="47da3bba-58fa-47a1-b7b5-cc9d2462c22c" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.245463 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d64291-d039-4acc-9240-6182267c1848" containerName="istio-proxy" Oct 30 20:14:03.250724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.245483 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cb227a-c86c-4838-80ef-756d2af0ff40" containerName="istio-proxy" Oct 30 20:14:03.333092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.333004 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:14:03.384729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.376105 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.409737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.409675 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.409914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.409897 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdtn\" (UniqueName: \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.433001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.431628 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bb565e\"" Oct 30 20:14:03.503864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.502663 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:14:03.532719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.521086 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdtn\" (UniqueName: \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.532719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.521539 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.566728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.559899 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.624265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.624186 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdtn\" (UniqueName: \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\") pod \"payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.634793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.634763 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:14:03.635283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.634899 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.690723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.689917 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bb565e-dockercfg-xdqp8\"" Oct 30 20:14:03.811172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.811121 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2n2\" (UniqueName: \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.811378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.811241 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.913826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.913780 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2n2\" (UniqueName: \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.914044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.913887 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.914317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.914292 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:03.915112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.915092 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:03.955067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.955031 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:14:03.960621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.960285 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:03.960621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:03.960357 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:04.023878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.022523 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2n2\" (UniqueName: \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\") pod \"payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:04.133399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.133364 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:04.133554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.133432 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:04.274565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.274480 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:04.416008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.414400 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:14:04.520942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.520901 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:14:04.527025 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.526944 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.549724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.545109 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55s7r\" (UniqueName: \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.549724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.545411 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.549724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.545548 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.549724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.545590 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.596725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.595345 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3736ef9-serving-cert\"" Oct 30 20:14:04.611729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.610927 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:14:04.694038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.693050 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55s7r\" (UniqueName: \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.694038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.693278 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.694038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.693376 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.694038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.693417 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.706832 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.705816 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.706832 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:04.705936 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-serving-cert: secret "maas-api-e2e-mt-f3736ef9-serving-cert" not found Oct 30 20:14:04.706832 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:04.706012 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls podName:48b890a8-5e32-4ddf-b802-0647ff7fe12a nodeName:}" failed. No retries permitted until 2026-10-30 20:14:05.205992872 +0000 UTC m=+1940.049664495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls") pod "maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" (UID: "48b890a8-5e32-4ddf-b802-0647ff7fe12a") : secret "maas-api-e2e-mt-f3736ef9-serving-cert" not found Oct 30 20:14:04.718399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.717641 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.778072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.752390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:04.824973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.810863 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55s7r\" (UniqueName: \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:04.998784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.984916 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:04.998784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:04.984982 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:05.040898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.034301 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.115036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.105829 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:05.115036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.105892 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:05.191186 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:14:05.187223 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6332187_0427_4050_b378_430da06c18af.slice/crio-6c3a07378a3f63d1d0e879ab9729c70266cfddc7baec41653cf853a85abb3edd WatchSource:0}: Error finding container 6c3a07378a3f63d1d0e879ab9729c70266cfddc7baec41653cf853a85abb3edd: Status 404 returned error can't find the container with id 6c3a07378a3f63d1d0e879ab9729c70266cfddc7baec41653cf853a85abb3edd Oct 30 20:14:05.191186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.188687 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:14:05.212826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.203684 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" event={"ID":"f338317f-bf2f-43d6-a8bc-2dfadebd4871","Type":"ContainerStarted","Data":"2dcb755802dcef77f8e46f7223e67025447ed36e62dda5bba201a7a77002148f"} Oct 30 20:14:05.212826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.203774 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" event={"ID":"f338317f-bf2f-43d6-a8bc-2dfadebd4871","Type":"ContainerStarted","Data":"4f4d5a7f9cb38718203bada0d6a0c3551e215dd8832b3afe0926504ffaf8da6e"} Oct 30 20:14:05.212826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.207334 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:05.318821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.298143 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.318821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.299364 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:05.440067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.440020 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:14:05.441162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.441118 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") pod \"maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:05.521277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.521212 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" podStartSLOduration=3.52119423 podStartE2EDuration="3.52119423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:05.506709305 +0000 UTC m=+1940.350380927" watchObservedRunningTime="2026-10-30 20:14:05.52119423 +0000 UTC m=+1940.364865860" Oct 30 20:14:05.659310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:05.641263 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:06.009864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.009742 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:06.009864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.009803 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:06.076914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.076869 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:06.077191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.077159 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:06.225832 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.225743 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" event={"ID":"a6332187-0427-4050-b378-430da06c18af","Type":"ContainerStarted","Data":"dd8034f281417ec3a11e7de13304e1fc80ed09edd3d1a596577b0400e70da057"} Oct 30 20:14:06.226112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.226090 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" event={"ID":"a6332187-0427-4050-b378-430da06c18af","Type":"ContainerStarted","Data":"6c3a07378a3f63d1d0e879ab9729c70266cfddc7baec41653cf853a85abb3edd"} Oct 30 20:14:06.226282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.226269 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:06.345847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.345784 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" podStartSLOduration=4.345763682 podStartE2EDuration="4.345763682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:06.341787423 +0000 UTC m=+1941.185459050" watchObservedRunningTime="2026-10-30 20:14:06.345763682 +0000 UTC m=+1941.189435312" Oct 30 20:14:06.612740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.606344 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:14:06.612740 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:14:06.612503 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b890a8_5e32_4ddf_b802_0647ff7fe12a.slice/crio-0fe9ea0dfd98553768d36eca599fb783470c83039b395b68c35d7cf6d18bd38a WatchSource:0}: Error finding container 0fe9ea0dfd98553768d36eca599fb783470c83039b395b68c35d7cf6d18bd38a: Status 404 returned error can't find the container with id 0fe9ea0dfd98553768d36eca599fb783470c83039b395b68c35d7cf6d18bd38a Oct 30 20:14:06.960724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.960301 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:06.960724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:06.960376 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:07.094723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:07.078820 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:07.094723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:07.078916 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:07.231304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:07.231211 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" event={"ID":"48b890a8-5e32-4ddf-b802-0647ff7fe12a","Type":"ContainerStarted","Data":"0fe9ea0dfd98553768d36eca599fb783470c83039b395b68c35d7cf6d18bd38a"} Oct 30 20:14:07.961722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:07.958972 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:07.961722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:07.959044 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:08.081753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.081312 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:14:08.081753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.081376 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:14:08.241079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.240978 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" event={"ID":"48b890a8-5e32-4ddf-b802-0647ff7fe12a","Type":"ContainerStarted","Data":"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5"} Oct 30 20:14:08.241571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.241074 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:08.352925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.345444 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:14:08.957722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.954308 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:14:08.957722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:08.954370 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 20:14:09.079315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:09.079280 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:14:09.080427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:09.080406 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:14:09.202897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:09.202833 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" podStartSLOduration=4.311551037 podStartE2EDuration="5.202814621s" totalImagesPullingTime="891.263584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:08.419507907 +0000 UTC m=+1943.263179537" watchObservedRunningTime="2026-10-30 20:14:09.202814621 +0000 UTC m=+1944.046486254" Oct 30 20:14:09.952527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:09.952492 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:14:09.953693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:09.953668 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:14:10.289816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:10.286287 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.523733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:10.510693 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.139853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:12.139815 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.360724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:12.360453 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.792319 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:12.792283 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:14:14.262606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:14.262574 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:14:15.321607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.321572 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:14:15.326231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.326203 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.386102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.386077 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:14:15.391446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.391424 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f44ec2-serving-cert\"" Oct 30 20:14:15.404687 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.404656 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.404846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.404829 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.404916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.404872 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.404980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.404955 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs46n\" (UniqueName: \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.506337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.506300 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.506501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.506362 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.506501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.506398 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs46n\" (UniqueName: \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.506501 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:15.506464 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-serving-cert: secret "maas-api-e2e-shared-a-w4-f44ec2-serving-cert" not found Oct 30 20:14:15.506687 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:15.506545 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls podName:92fdeca4-c61c-4c03-9044-79186985b53c nodeName:}" failed. No retries permitted until 2026-10-30 20:14:16.00652484 +0000 UTC m=+1950.850196461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls") pod "maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" (UID: "92fdeca4-c61c-4c03-9044-79186985b53c") : secret "maas-api-e2e-shared-a-w4-f44ec2-serving-cert" not found Oct 30 20:14:15.506853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.506821 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.507621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.507596 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.509100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.509078 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.657736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.656692 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs46n\" (UniqueName: \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:15.668720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.664280 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.904302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:15.904246 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:16.029781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.029674 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:16.032266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.032239 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:16.174600 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.174563 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:16.240355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.240322 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:16.413844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.413795 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:16.638054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.638021 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:14:16.638284 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:14:16.638255 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fdeca4_c61c_4c03_9044_79186985b53c.slice/crio-c77aaee42cafe278d0031697703470d40d8084d76569b63677415b8d438bc458 WatchSource:0}: Error finding container c77aaee42cafe278d0031697703470d40d8084d76569b63677415b8d438bc458: Status 404 returned error can't find the container with id c77aaee42cafe278d0031697703470d40d8084d76569b63677415b8d438bc458 Oct 30 20:14:16.639949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:16.639929 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:14:17.234886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:17.234858 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:14:17.320770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:17.320739 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" event={"ID":"92fdeca4-c61c-4c03-9044-79186985b53c","Type":"ContainerStarted","Data":"c77aaee42cafe278d0031697703470d40d8084d76569b63677415b8d438bc458"} Oct 30 20:14:18.245024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:18.244995 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:14:18.331943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:18.331901 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" event={"ID":"92fdeca4-c61c-4c03-9044-79186985b53c","Type":"ContainerStarted","Data":"0641a75ed04387478e968a09a1f20e73dcaba1dd15400893eb7e10b95243b878"} Oct 30 20:14:18.332134 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:18.332044 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:18.500430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:18.500333 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" podStartSLOduration=2.661445751 podStartE2EDuration="3.500318042s" totalImagesPullingTime="838.872291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:18.497750539 +0000 UTC m=+1953.341422165" watchObservedRunningTime="2026-10-30 20:14:18.500318042 +0000 UTC m=+1953.343989672" Oct 30 20:14:19.732637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:19.732592 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:19.945426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:19.945393 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:20.338061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.337756 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:14:20.550721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.401309 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.576720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.427129 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:14:20.577313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.577284 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:14:20.581728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.447989 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98sr\" (UniqueName: \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.581728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.581409 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.615791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.612491 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-77fcfa\"" Oct 30 20:14:20.671892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.671844 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-77fcfa-dockercfg-k4vmb\"" Oct 30 20:14:20.681808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.677873 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:14:20.681808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.678081 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.712721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.705929 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.712721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.706212 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98sr\" (UniqueName: \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.712721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.707760 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.797170 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.797120 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98sr\" (UniqueName: \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\") pod \"payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.812154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.812111 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjm9\" (UniqueName: \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.812525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.812497 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.922727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.916075 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjm9\" (UniqueName: \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.922727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.916747 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.922727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.918248 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:20.959014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.958979 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:20.977614 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:20.977579 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjm9\" (UniqueName: \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\") pod \"payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:21.042534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.042502 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:21.345158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.345127 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:14:21.345288 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:14:21.345251 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60306054_eae8_4fb9_b1b1_7841ff67ebb6.slice/crio-ea12c6bbaf28715d34bf74319c71f412339af39ee0f6610f3013c7d5cd58d93c WatchSource:0}: Error finding container ea12c6bbaf28715d34bf74319c71f412339af39ee0f6610f3013c7d5cd58d93c: Status 404 returned error can't find the container with id ea12c6bbaf28715d34bf74319c71f412339af39ee0f6610f3013c7d5cd58d93c Oct 30 20:14:21.454664 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.454642 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:14:21.455184 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:14:21.455165 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8a200a_3f8b_4daf_9728_506fc78ed149.slice/crio-97425f88e2410b3cee8de0eeda9da2c6d3915f86c6b44e98ae6b4be1807615d7 WatchSource:0}: Error finding container 97425f88e2410b3cee8de0eeda9da2c6d3915f86c6b44e98ae6b4be1807615d7: Status 404 returned error can't find the container with id 97425f88e2410b3cee8de0eeda9da2c6d3915f86c6b44e98ae6b4be1807615d7 Oct 30 20:14:21.468073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.467496 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" event={"ID":"60306054-eae8-4fb9-b1b1-7841ff67ebb6","Type":"ContainerStarted","Data":"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb"} Oct 30 20:14:21.468073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.467563 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:21.468073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.467590 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" event={"ID":"60306054-eae8-4fb9-b1b1-7841ff67ebb6","Type":"ContainerStarted","Data":"ea12c6bbaf28715d34bf74319c71f412339af39ee0f6610f3013c7d5cd58d93c"} Oct 30 20:14:21.468707 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.468674 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" event={"ID":"2b8a200a-3f8b-4daf-9728-506fc78ed149","Type":"ContainerStarted","Data":"97425f88e2410b3cee8de0eeda9da2c6d3915f86c6b44e98ae6b4be1807615d7"} Oct 30 20:14:21.586352 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.586306 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" podStartSLOduration=1.586291616 podStartE2EDuration="1.586291616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:21.582896354 +0000 UTC m=+1956.426567984" watchObservedRunningTime="2026-10-30 20:14:21.586291616 +0000 UTC m=+1956.429963244" Oct 30 20:14:21.845966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:21.845925 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:22.476755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:22.476717 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" event={"ID":"2b8a200a-3f8b-4daf-9728-506fc78ed149","Type":"ContainerStarted","Data":"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d"} Oct 30 20:14:22.477044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:22.477027 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:22.596170 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:22.596118 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" podStartSLOduration=2.596102921 podStartE2EDuration="2.596102921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:22.592214831 +0000 UTC m=+1957.435886490" watchObservedRunningTime="2026-10-30 20:14:22.596102921 +0000 UTC m=+1957.439774550" Oct 30 20:14:24.344555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:24.344526 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:14:26.747727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:26.747544 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.582723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:31.578824 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.480404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:32.480375 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:14:33.488240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:33.488207 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:14:36.879040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:36.878997 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:41.978253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:41.978210 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.797163 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:42.797122 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:14:43.840860 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.840780 4524 generic.go:356] "Generic (PLEG): container finished" podID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerID="76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f" exitCode=0 Oct 30 20:14:43.840860 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.840840 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerDied","Data":"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f"} Oct 30 20:14:43.841398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.841247 4524 scope.go:122] "RemoveContainer" containerID="76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f" Oct 30 20:14:43.843041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.843014 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:43.843162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.843083 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:43.843162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:43.843126 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:14:44.857724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:44.856911 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerStarted","Data":"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b"} Oct 30 20:14:45.469721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:45.463005 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:14:45.473718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:45.470930 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:45.473718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:45.470993 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:45.581063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:45.580900 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:46.469719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:46.468465 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:46.469719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:46.468548 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:47.461175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:47.461133 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:47.462198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:47.462162 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:47.554009 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:47.553893 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.461341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:48.461304 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:48.461639 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:48.461610 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:49.464101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:49.464060 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:49.465501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:49.465469 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:50.460534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.460502 4524 patch_prober.go:29] interesting pod/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:14:50.460740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.460560 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:14:50.485553 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.485503 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:14:50.486096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.485779 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" containerID="cri-o://2ff8654519e5d2c1bf630db1dbd290e27fa7fd49c1b579c40845b7aa692b2256" gracePeriod=30 Oct 30 20:14:50.489917 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.489890 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:14:50.490090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.490069 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" containerID="cri-o://63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef" gracePeriod=30 Oct 30 20:14:50.537349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.537308 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:14:50.537631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.537602 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" containerID="cri-o://19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666" gracePeriod=30 Oct 30 20:14:50.543480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.543448 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:14:50.543639 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.543601 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" containerID="cri-o://61c62cac9a92d205b3d83ccbd7292812ba5199ef90aa15bcffa78828c7fcf98a" gracePeriod=30 Oct 30 20:14:50.970836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:50.964006 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:51.466242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:51.466210 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:14:51.466817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:51.466781 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:14:51.467633 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:51.467612 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:14:52.257746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:52.257661 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.719391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:52.718380 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:52.719391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:52.718454 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:53.214625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:53.214555 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.950949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:53.950909 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:53.951333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:53.950993 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:54.812745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:54.812269 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:55.027671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:55.027639 4524 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:55.028062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:55.027734 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:55.274362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:55.274263 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:55.979906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:55.979507 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:14:55.994277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:55.994147 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:14:56.059477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.059443 4524 generic.go:356] "Generic (PLEG): container finished" podID="95da0860-06ba-431f-87d7-d77554fd280d" containerID="2ff8654519e5d2c1bf630db1dbd290e27fa7fd49c1b579c40845b7aa692b2256" exitCode=0 Oct 30 20:14:56.060073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.059591 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" event={"ID":"95da0860-06ba-431f-87d7-d77554fd280d","Type":"ContainerDied","Data":"2ff8654519e5d2c1bf630db1dbd290e27fa7fd49c1b579c40845b7aa692b2256"} Oct 30 20:14:56.061769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.061740 4524 generic.go:356] "Generic (PLEG): container finished" podID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerID="61c62cac9a92d205b3d83ccbd7292812ba5199ef90aa15bcffa78828c7fcf98a" exitCode=0 Oct 30 20:14:56.061982 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.061799 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" event={"ID":"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641","Type":"ContainerDied","Data":"61c62cac9a92d205b3d83ccbd7292812ba5199ef90aa15bcffa78828c7fcf98a"} Oct 30 20:14:56.064108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.064087 4524 generic.go:356] "Generic (PLEG): container finished" podID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerID="19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666" exitCode=0 Oct 30 20:14:56.064354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.064209 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" Oct 30 20:14:56.064354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.064209 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" event={"ID":"cd340e01-9c01-48e1-a732-89bbb7440cc0","Type":"ContainerDied","Data":"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666"} Oct 30 20:14:56.064354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.064338 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc" event={"ID":"cd340e01-9c01-48e1-a732-89bbb7440cc0","Type":"ContainerDied","Data":"681cfa3e4313250138074630e0d90f1bbdf68199281aaddcd503aec6afc86509"} Oct 30 20:14:56.064615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.064374 4524 scope.go:122] "RemoveContainer" containerID="19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666" Oct 30 20:14:56.066107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.066084 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:14:56.066232 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.066187 4524 generic.go:356] "Generic (PLEG): container finished" podID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerID="63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef" exitCode=0 Oct 30 20:14:56.066297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.066252 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" event={"ID":"fa4bccdc-1419-4643-bcef-55dc576f1bad","Type":"ContainerDied","Data":"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef"} Oct 30 20:14:56.066297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.066289 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" event={"ID":"fa4bccdc-1419-4643-bcef-55dc576f1bad","Type":"ContainerDied","Data":"23a9cdc812282ada1cb3c24006829ebedb8d8d7db1a0035f14dcda0ec4730f3e"} Oct 30 20:14:56.066460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.066422 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5" Oct 30 20:14:56.076000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.075978 4524 scope.go:122] "RemoveContainer" containerID="19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666" Oct 30 20:14:56.076311 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:56.076286 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666\": container with ID starting with 19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666 not found: ID does not exist" containerID="19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666" Oct 30 20:14:56.076396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.076317 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666"} err="failed to get container status \"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666\": rpc error: code = NotFound desc = could not find container \"19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666\": container with ID starting with 19ac8b43926314cb869edea3d8c6429f309ec1c70d8698b73f3ba81a8eac1666 not found: ID does not exist" Oct 30 20:14:56.076396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.076334 4524 scope.go:122] "RemoveContainer" containerID="63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef" Oct 30 20:14:56.087347 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.087332 4524 scope.go:122] "RemoveContainer" containerID="63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef" Oct 30 20:14:56.087591 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:14:56.087572 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef\": container with ID starting with 63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef not found: ID does not exist" containerID="63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef" Oct 30 20:14:56.087659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.087602 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef"} err="failed to get container status \"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef\": rpc error: code = NotFound desc = could not find container \"63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef\": container with ID starting with 63d0dd56ad26098c5ef4d8153825a85a09a997310e8d183daada7fa8a7f239ef not found: ID does not exist" Oct 30 20:14:56.088254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.088234 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:14:56.105075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105005 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105070 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105106 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105152 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105199 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105280 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.105415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105294 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105367 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.105620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105417 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105448 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105588 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105633 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105671 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.105815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105756 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.105815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105798 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105805 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105828 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105832 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105865 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105887 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\") pod \"cd340e01-9c01-48e1-a732-89bbb7440cc0\" (UID: \"cd340e01-9c01-48e1-a732-89bbb7440cc0\") " Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.105934 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.106024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106016 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\") pod \"fa4bccdc-1419-4643-bcef-55dc576f1bad\" (UID: \"fa4bccdc-1419-4643-bcef-55dc576f1bad\") " Oct 30 20:14:56.106392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106116 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.106392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106130 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.106500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106425 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.106984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106959 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4bccdc-1419-4643-bcef-55dc576f1bad-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.106987 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd340e01-9c01-48e1-a732-89bbb7440cc0-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.107002 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.107016 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.107031 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.107045 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.107090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.107059 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.108395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.108097 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "kube-api-access-49f5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.108395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.108354 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.108510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.108413 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.108510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.108443 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.108886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.108863 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.109321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.109298 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.109943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.109510 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "kube-api-access-mgxvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.109943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.109549 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:56.111000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.110798 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy" pod "fa4bccdc-1419-4643-bcef-55dc576f1bad" (UID: "fa4bccdc-1419-4643-bcef-55dc576f1bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.111111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.111048 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:56.111216 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.111193 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy" pod "cd340e01-9c01-48e1-a732-89bbb7440cc0" (UID: "cd340e01-9c01-48e1-a732-89bbb7440cc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.208276 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208242 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.208440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208287 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.208440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208332 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.208440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208369 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.208622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208444 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.208622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208479 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.208622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208577 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.208622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208615 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.208881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208645 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.208881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208649 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.208881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208691 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.208881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208817 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208890 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.208930 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209029 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209057 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209122 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\") pod \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\" (UID: \"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209175 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209179 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209247 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\") pod \"95da0860-06ba-431f-87d7-d77554fd280d\" (UID: \"95da0860-06ba-431f-87d7-d77554fd280d\") " Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209297 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209380 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209400 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209517 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209799 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.209981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.209820 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210338 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210363 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210378 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210392 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210406 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210419 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210432 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4bccdc-1419-4643-bcef-55dc576f1bad-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210446 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210458 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210472 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxvt\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-kube-api-access-mgxvt\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210486 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210499 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95da0860-06ba-431f-87d7-d77554fd280d-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210513 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210526 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f5k\" (UniqueName: \"kubernetes.io/projected/fa4bccdc-1419-4643-bcef-55dc576f1bad-kube-api-access-49f5k\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210540 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210554 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210569 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210583 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210597 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd340e01-9c01-48e1-a732-89bbb7440cc0-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.210847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.210657 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:56.212045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.211323 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.212443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.212414 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:56.212547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.212498 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.213541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.213512 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "kube-api-access-rs7n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.214304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.214274 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "kube-api-access-k7z2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.214499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.214472 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.214587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.214501 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo" pod "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" (UID: "54c2a6ab-82ca-4b50-98d0-7d25ca8b3641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:56.214728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.214677 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.215304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.215278 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy" pod "95da0860-06ba-431f-87d7-d77554fd280d" (UID: "95da0860-06ba-431f-87d7-d77554fd280d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:56.312628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.312581 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.312922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.312899 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7z2n\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-kube-api-access-k7z2n\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313026 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313168 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313300 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95da0860-06ba-431f-87d7-d77554fd280d-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313414 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95da0860-06ba-431f-87d7-d77554fd280d-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313553 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313540 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs7n6\" (UniqueName: \"kubernetes.io/projected/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-kube-api-access-rs7n6\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313651 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313640 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313774 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.313923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.313893 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95da0860-06ba-431f-87d7-d77554fd280d-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.745116 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.745079 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:14:56.749190 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.749161 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k7rx5"] Oct 30 20:14:56.857591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.857562 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:14:56.861183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:56.861163 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-tdqqc"] Oct 30 20:14:57.075302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.075224 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" Oct 30 20:14:57.075302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.075247 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm" event={"ID":"95da0860-06ba-431f-87d7-d77554fd280d","Type":"ContainerDied","Data":"a693af19969cdd779cbb7ef85c17a514a24e8dda76d61e2d42e62065f9500ad7"} Oct 30 20:14:57.075846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.075351 4524 scope.go:122] "RemoveContainer" containerID="2ff8654519e5d2c1bf630db1dbd290e27fa7fd49c1b579c40845b7aa692b2256" Oct 30 20:14:57.077365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.077341 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" Oct 30 20:14:57.077742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.077684 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc" event={"ID":"54c2a6ab-82ca-4b50-98d0-7d25ca8b3641","Type":"ContainerDied","Data":"e8f3f8359fb6f359cc2ae070740c71095df8b474e672f95ae82ff5e2178d6574"} Oct 30 20:14:57.087825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.087809 4524 scope.go:122] "RemoveContainer" containerID="61c62cac9a92d205b3d83ccbd7292812ba5199ef90aa15bcffa78828c7fcf98a" Oct 30 20:14:57.247688 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.247656 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:14:57.253424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.253394 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-t6mnc"] Oct 30 20:14:57.301791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.301764 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" path="/var/lib/kubelet/pods/54c2a6ab-82ca-4b50-98d0-7d25ca8b3641/volumes" Oct 30 20:14:57.302335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.302321 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" path="/var/lib/kubelet/pods/cd340e01-9c01-48e1-a732-89bbb7440cc0/volumes" Oct 30 20:14:57.302839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.302823 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" path="/var/lib/kubelet/pods/fa4bccdc-1419-4643-bcef-55dc576f1bad/volumes" Oct 30 20:14:57.366430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.365879 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:14:57.371204 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:57.371179 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-knxfm"] Oct 30 20:14:58.629305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:58.629272 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:58.909723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:58.904915 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:59.307691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:14:59.307617 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95da0860-06ba-431f-87d7-d77554fd280d" path="/var/lib/kubelet/pods/95da0860-06ba-431f-87d7-d77554fd280d/volumes" Oct 30 20:15:02.360273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.360233 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:02.637783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.636756 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.810762 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.811920 4524 cpu_manager.go:414] "containerMap: removing container" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.811943 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812024 4524 cpu_manager.go:414] "containerMap: removing container" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812034 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812052 4524 cpu_manager.go:414] "containerMap: removing container" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812061 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812088 4524 cpu_manager.go:414] "containerMap: removing container" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812097 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812320 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="95da0860-06ba-431f-87d7-d77554fd280d" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812338 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd340e01-9c01-48e1-a732-89bbb7440cc0" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812351 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c2a6ab-82ca-4b50-98d0-7d25ca8b3641" containerName="istio-proxy" Oct 30 20:15:02.814726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.812365 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4bccdc-1419-4643-bcef-55dc576f1bad" containerName="istio-proxy" Oct 30 20:15:02.835485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.819195 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:02.850718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.836291 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:15:02.900231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.899778 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5d2c70-dockercfg-jm6fv\"" Oct 30 20:15:02.900456 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.899853 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5d2c70\"" Oct 30 20:15:02.914736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.902952 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:15:02.946149 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:02.946113 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.045925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.045898 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:15:03.048310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.047657 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vb2m\" (UniqueName: \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.048310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.047815 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.048310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.047999 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.048310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.048150 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6jt\" (UniqueName: \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.200843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.200796 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6jt\" (UniqueName: \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.201162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.201136 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vb2m\" (UniqueName: \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.201393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.201377 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.201647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.201631 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.202757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.202732 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.204113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.204089 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.283531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.283119 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6jt\" (UniqueName: \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\") pod \"payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.295731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.283857 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vb2m\" (UniqueName: \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\") pod \"payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.308604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.307993 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:03.394343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.394087 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:03.766771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.746049 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:15:03.908222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:03.908185 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:15:04.264376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.264341 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" event={"ID":"8640f882-eb60-4ee3-a0f1-2940dff09143","Type":"ContainerStarted","Data":"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116"} Oct 30 20:15:04.264549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.264400 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" event={"ID":"8640f882-eb60-4ee3-a0f1-2940dff09143","Type":"ContainerStarted","Data":"fb39e9cdd49197f9a5d4d78f447a3191b4eae719f899f672e593ca88d919d703"} Oct 30 20:15:04.264602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.264545 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:04.266162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.266139 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" event={"ID":"da281092-bd97-41ef-a470-a47e285a7da4","Type":"ContainerStarted","Data":"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a"} Oct 30 20:15:04.266296 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.266177 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" event={"ID":"da281092-bd97-41ef-a470-a47e285a7da4","Type":"ContainerStarted","Data":"5b160d77b1f2cf54c376b0e2646e714ebbe80fcea74ed17cf470fa2afa0eddb2"} Oct 30 20:15:04.266296 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.266231 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:04.384642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.384525 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" podStartSLOduration=2.384507459 podStartE2EDuration="2.384507459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.381227133 +0000 UTC m=+1999.224898761" watchObservedRunningTime="2026-10-30 20:15:04.384507459 +0000 UTC m=+1999.228179091" Oct 30 20:15:04.919725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:04.909174 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:06.905160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.905108 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" podStartSLOduration=4.905091025 podStartE2EDuration="4.905091025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.517916152 +0000 UTC m=+1999.361587783" watchObservedRunningTime="2026-10-30 20:15:06.905091025 +0000 UTC m=+2001.748762655" Oct 30 20:15:06.906464 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.906429 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:15:06.907158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.906687 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" containerID="cri-o://8568cd761d0d9f0c9f433b4fddde9ff6edfee715e4581052374c2a6d4cffa9ba" gracePeriod=30 Oct 30 20:15:06.957943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.957911 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:15:06.963531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.963470 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:15:06.963743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.963694 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" containerID="cri-o://d93002bc151f415f3e6dcb6b524433fd6a9a7d35030d1aeadf9deb96c43ae977" gracePeriod=30 Oct 30 20:15:06.963959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.963920 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" containerID="cri-o://8b38168b8209e515cd79886e0daa1cbfd0c439137145631f3e4f0a3f139cc711" gracePeriod=30 Oct 30 20:15:06.970061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.970034 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:15:06.970256 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:06.970234 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" containerID="cri-o://e5ad2e78bc18c46d7f04cc4aa1a053b65a0ba4213b8b9d310ee965d053cdf61d" gracePeriod=30 Oct 30 20:15:08.581202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:08.574648 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:08.581202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:08.574738 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:08.660295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:08.660258 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.251821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:10.246284 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.490895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:10.490857 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:10.491060 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:10.490914 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:10.928317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:10.928267 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.008766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:11.005660 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:11.008766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:11.005739 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:11.044725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:11.044212 4524 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:11.044725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:11.044274 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:11.206331 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:11.206180 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:12.033878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.025053 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.710968 4524 generic.go:356] "Generic (PLEG): container finished" podID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerID="d93002bc151f415f3e6dcb6b524433fd6a9a7d35030d1aeadf9deb96c43ae977" exitCode=0 Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.711201 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" event={"ID":"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1","Type":"ContainerDied","Data":"d93002bc151f415f3e6dcb6b524433fd6a9a7d35030d1aeadf9deb96c43ae977"} Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.714276 4524 generic.go:356] "Generic (PLEG): container finished" podID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerID="8b38168b8209e515cd79886e0daa1cbfd0c439137145631f3e4f0a3f139cc711" exitCode=0 Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.714398 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" event={"ID":"193a8f69-caf0-4429-9d15-f7f3077e5f07","Type":"ContainerDied","Data":"8b38168b8209e515cd79886e0daa1cbfd0c439137145631f3e4f0a3f139cc711"} Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.716978 4524 generic.go:356] "Generic (PLEG): container finished" podID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerID="8568cd761d0d9f0c9f433b4fddde9ff6edfee715e4581052374c2a6d4cffa9ba" exitCode=0 Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.717106 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" event={"ID":"914be0c0-72ee-4b2e-9d47-4abaf5fadef4","Type":"ContainerDied","Data":"8568cd761d0d9f0c9f433b4fddde9ff6edfee715e4581052374c2a6d4cffa9ba"} Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.719473 4524 generic.go:356] "Generic (PLEG): container finished" podID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerID="e5ad2e78bc18c46d7f04cc4aa1a053b65a0ba4213b8b9d310ee965d053cdf61d" exitCode=0 Oct 30 20:15:12.720114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.719609 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" event={"ID":"8c605388-39c6-4ba7-8938-528e4ed23b91","Type":"ContainerDied","Data":"e5ad2e78bc18c46d7f04cc4aa1a053b65a0ba4213b8b9d310ee965d053cdf61d"} Oct 30 20:15:12.873053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.870045 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:15:12.892343 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:12.892305 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:15:12.920560 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920533 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.920736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920610 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.920808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920783 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.920863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920820 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.920928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920860 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.921011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.920993 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.921069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921047 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.921121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921075 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.921176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921106 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:12.921236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921177 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\") pod \"8c605388-39c6-4ba7-8938-528e4ed23b91\" (UID: \"8c605388-39c6-4ba7-8938-528e4ed23b91\") " Oct 30 20:15:12.921732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921664 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:12.921973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921936 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c605388-39c6-4ba7-8938-528e4ed23b91-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:12.921973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.921959 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:12.922103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.922033 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:12.922335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.922299 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:12.922479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.922394 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:12.923652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.923618 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:12.923931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.923910 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:12.924821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.924794 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "kube-api-access-r9hqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:12.924923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:12.924827 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token" pod "8c605388-39c6-4ba7-8938-528e4ed23b91" (UID: "8c605388-39c6-4ba7-8938-528e4ed23b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026211 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026251 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026267 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026286 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9hqr\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-kube-api-access-r9hqr\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026303 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026316 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c605388-39c6-4ba7-8938-528e4ed23b91-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.030748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.026331 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c605388-39c6-4ba7-8938-528e4ed23b91-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.073220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.073178 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:15:13.126858 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.126829 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.126984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.126964 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127003 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127093 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127196 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127234 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127248 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.127376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127316 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127382 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127431 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\") pod \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\" (UID: \"914be0c0-72ee-4b2e-9d47-4abaf5fadef4\") " Oct 30 20:15:13.127476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127439 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.127676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127652 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.127769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127677 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.127875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.127848 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:13.128350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.128326 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.128453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.128357 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.128453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.128372 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.128453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.128386 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.128453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.128401 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.129446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.129408 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.129535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.129454 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:13.129535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.129518 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.129791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.129774 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:15:13.129855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.129783 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl" pod "914be0c0-72ee-4b2e-9d47-4abaf5fadef4" (UID: "914be0c0-72ee-4b2e-9d47-4abaf5fadef4"). InnerVolumeSpecName "kube-api-access-jh5nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.133903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.133886 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:15:13.229754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229721 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.229934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229790 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.229934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229852 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.229934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229909 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229944 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.229981 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.230110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230034 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230109 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230148 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.230285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230166 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230254 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230285 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230325 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230356 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230382 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230416 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.230466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230459 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230517 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230615 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230624 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.230686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230650 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.230686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230670 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:13.230961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230736 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\") pod \"193a8f69-caf0-4429-9d15-f7f3077e5f07\" (UID: \"193a8f69-caf0-4429-9d15-f7f3077e5f07\") " Oct 30 20:15:13.230961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.230785 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\") pod \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\" (UID: \"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1\") " Oct 30 20:15:13.231244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231182 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:13.231742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231457 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.231861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231803 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.231861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231830 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.231861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231848 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231863 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231880 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5nl\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-kube-api-access-jh5nl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231897 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231912 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193a8f69-caf0-4429-9d15-f7f3077e5f07-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231927 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231941 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231955 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231970 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.231985 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914be0c0-72ee-4b2e-9d47-4abaf5fadef4-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.232599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.232565 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.232762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.232692 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "kube-api-access-h2rrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.232894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.232869 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.232984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.232965 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.233043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.233012 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:13.233230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.233205 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "kube-api-access-5wpr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.233302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.233277 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:13.233549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.233532 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token" pod "193a8f69-caf0-4429-9d15-f7f3077e5f07" (UID: "193a8f69-caf0-4429-9d15-f7f3077e5f07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.234478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.234444 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:13.234478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.234455 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token" pod "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" (UID: "f2fe1cfe-720c-4c29-b5d9-f2b109d689f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:13.236329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.236267 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.333213 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333171 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333213 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333211 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333213 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333221 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wpr9\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-kube-api-access-5wpr9\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333232 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333243 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333251 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333260 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193a8f69-caf0-4429-9d15-f7f3077e5f07-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333269 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333277 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rrt\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-kube-api-access-h2rrt\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.333390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.333285 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193a8f69-caf0-4429-9d15-f7f3077e5f07-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:13.987721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.978938 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" event={"ID":"f2fe1cfe-720c-4c29-b5d9-f2b109d689f1","Type":"ContainerDied","Data":"e8c914d05627bf08482f5126f6fbb1f85935a5b249719db2bd8462a5f308da69"} Oct 30 20:15:13.988518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.988488 4524 scope.go:122] "RemoveContainer" containerID="d93002bc151f415f3e6dcb6b524433fd6a9a7d35030d1aeadf9deb96c43ae977" Oct 30 20:15:13.988841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:13.980260 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt" Oct 30 20:15:14.033021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.032118 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" Oct 30 20:15:14.033021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.032143 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g" event={"ID":"193a8f69-caf0-4429-9d15-f7f3077e5f07","Type":"ContainerDied","Data":"55d783008da8b9d9e161f67b7f581c6657c9256cb2259e7a12398f9563ea305d"} Oct 30 20:15:14.038413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.038392 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" Oct 30 20:15:14.039371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.038464 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq" event={"ID":"914be0c0-72ee-4b2e-9d47-4abaf5fadef4","Type":"ContainerDied","Data":"16af2bf60f9a05cf7ef5a7ea522b91da79f2d47db34165e7985a75e64fc3f4bc"} Oct 30 20:15:14.083677 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.083638 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" event={"ID":"8c605388-39c6-4ba7-8938-528e4ed23b91","Type":"ContainerDied","Data":"16dc8f6a05326b4b52b545dcc44860a060a73a73873423b53fe130b11433788b"} Oct 30 20:15:14.084315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.084290 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8" Oct 30 20:15:14.111875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.111849 4524 scope.go:122] "RemoveContainer" containerID="8b38168b8209e515cd79886e0daa1cbfd0c439137145631f3e4f0a3f139cc711" Oct 30 20:15:14.131856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.131830 4524 scope.go:122] "RemoveContainer" containerID="8568cd761d0d9f0c9f433b4fddde9ff6edfee715e4581052374c2a6d4cffa9ba" Oct 30 20:15:14.145672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.145650 4524 scope.go:122] "RemoveContainer" containerID="e5ad2e78bc18c46d7f04cc4aa1a053b65a0ba4213b8b9d310ee965d053cdf61d" Oct 30 20:15:14.235293 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.235223 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:15:14.285176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.285146 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-nbk4g"] Oct 30 20:15:14.355630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.355596 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:15:14.362770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.362743 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-knbkt"] Oct 30 20:15:14.475771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.475714 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:15:14.480596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.480572 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-5kzcq"] Oct 30 20:15:14.591297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.591218 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:15:14.595349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:14.595320 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-rtrl8"] Oct 30 20:15:15.292723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.286369 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:15:15.396876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.396839 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" path="/var/lib/kubelet/pods/193a8f69-caf0-4429-9d15-f7f3077e5f07/volumes" Oct 30 20:15:15.397420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.397405 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" path="/var/lib/kubelet/pods/8c605388-39c6-4ba7-8938-528e4ed23b91/volumes" Oct 30 20:15:15.397900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.397885 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" path="/var/lib/kubelet/pods/914be0c0-72ee-4b2e-9d47-4abaf5fadef4/volumes" Oct 30 20:15:15.398694 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.398678 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" path="/var/lib/kubelet/pods/f2fe1cfe-720c-4c29-b5d9-f2b109d689f1/volumes" Oct 30 20:15:15.399163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.399141 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:15:15.725720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:15.725140 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.372605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.372566 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373350 4524 cpu_manager.go:414] "containerMap: removing container" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373373 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373386 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373391 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373402 4524 cpu_manager.go:414] "containerMap: removing container" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373407 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373449 4524 cpu_manager.go:414] "containerMap: removing container" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" Oct 30 20:15:20.373493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373455 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" Oct 30 20:15:20.374141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373568 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fe1cfe-720c-4c29-b5d9-f2b109d689f1" containerName="istio-proxy" Oct 30 20:15:20.374141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373577 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c605388-39c6-4ba7-8938-528e4ed23b91" containerName="istio-proxy" Oct 30 20:15:20.374141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373588 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="193a8f69-caf0-4429-9d15-f7f3077e5f07" containerName="istio-proxy" Oct 30 20:15:20.374141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.373601 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="914be0c0-72ee-4b2e-9d47-4abaf5fadef4" containerName="istio-proxy" Oct 30 20:15:20.378809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.378784 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.435346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.435314 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:15:20.477750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.477686 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bb565e-serving-cert\"" Oct 30 20:15:20.494659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.494631 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dls2\" (UniqueName: \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.494906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.494670 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.495040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.495020 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.495263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.495241 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.597039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.597003 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.597231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.597152 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.597302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.597293 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dls2\" (UniqueName: \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.597359 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.597322 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.597584 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:20.597555 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-serving-cert: secret "maas-api-e2e-shared-b-w3-bb565e-serving-cert" not found Oct 30 20:15:20.597740 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:20.597654 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls podName:8c957d15-c365-4aae-a8de-b34d0ec66635 nodeName:}" failed. No retries permitted until 2026-10-30 20:15:21.097632007 +0000 UTC m=+2015.941303614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls") pod "maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" (UID: "8c957d15-c365-4aae-a8de-b34d0ec66635") : secret "maas-api-e2e-shared-b-w3-bb565e-serving-cert" not found Oct 30 20:15:20.597822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.597771 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.599805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.599782 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.668743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.668687 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dls2\" (UniqueName: \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:20.716275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.716232 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.980810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.980079 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.994917 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.994887 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:15:20.995373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:20.995341 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" containerID="cri-o://56803420a2845ca4a772f3d3c1334e9780c56f1e3e86506e56c85bb0e1b105c8" gracePeriod=30 Oct 30 20:15:21.003404 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.003379 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 20:15:21.003826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.003795 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" containerID="cri-o://517ff5474451a592581b385f3b44b739781b900191f36818c58ce5568863f15a" gracePeriod=30 Oct 30 20:15:21.007780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.007757 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:15:21.008199 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.008173 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" containerID="cri-o://8c1eeb668ff0a034ed1d919b4d261fe6ca58e5605169016439e8ff0075e2578b" gracePeriod=30 Oct 30 20:15:21.011332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.011246 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:15:21.011437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.011410 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" containerID="cri-o://9267fe4502441f6ce5e721d306b21e8cbdd4c787f9c1e47718efde87aef860a2" gracePeriod=30 Oct 30 20:15:21.109358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.109320 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:21.112226 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.112199 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:21.295431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.295336 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:21.679151 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:15:21.679121 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c957d15_c365_4aae_a8de_b34d0ec66635.slice/crio-d08e710a2a57ef66376f6c4fd37e0b30f52f527e95b9ac40a54605593803352d WatchSource:0}: Error finding container d08e710a2a57ef66376f6c4fd37e0b30f52f527e95b9ac40a54605593803352d: Status 404 returned error can't find the container with id d08e710a2a57ef66376f6c4fd37e0b30f52f527e95b9ac40a54605593803352d Oct 30 20:15:21.679953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.679928 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:15:21.759673 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.759640 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:15:21.759942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.759914 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" containerID="cri-o://825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122" gracePeriod=30 Oct 30 20:15:21.763565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.763539 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:15:21.763793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.763759 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" containerID="cri-o://61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3" gracePeriod=30 Oct 30 20:15:21.766797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.766752 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:15:21.766976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.766948 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" containerID="cri-o://c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a" gracePeriod=30 Oct 30 20:15:21.769863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.769841 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 20:15:21.770026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:21.770005 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" containerID="cri-o://9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f" gracePeriod=30 Oct 30 20:15:22.217400 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:22.217365 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" event={"ID":"8c957d15-c365-4aae-a8de-b34d0ec66635","Type":"ContainerStarted","Data":"d08e710a2a57ef66376f6c4fd37e0b30f52f527e95b9ac40a54605593803352d"} Oct 30 20:15:22.275234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:22.275201 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:15:22.275424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:22.275400 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" containerID="cri-o://e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2" gracePeriod=30 Oct 30 20:15:23.225709 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:23.225662 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" event={"ID":"8c957d15-c365-4aae-a8de-b34d0ec66635","Type":"ContainerStarted","Data":"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e"} Oct 30 20:15:23.226168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:23.225936 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:23.344943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:23.344885 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" podStartSLOduration=2.4540710470000002 podStartE2EDuration="3.344866376s" totalImagesPullingTime="890.795329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:23.343608331 +0000 UTC m=+2018.187279959" watchObservedRunningTime="2026-10-30 20:15:23.344866376 +0000 UTC m=+2018.188538006" Oct 30 20:15:25.094449 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:25.094415 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:25.094856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:25.094477 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:25.831015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:25.830976 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.929250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:25.928793 4524 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:25.929250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:25.928859 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:26.067866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.067835 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:15:26.067962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.067897 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:15:26.094870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.094798 4524 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:15:26.095260 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.094887 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:15:26.258094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.258064 4524 generic.go:356] "Generic (PLEG): container finished" podID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerID="517ff5474451a592581b385f3b44b739781b900191f36818c58ce5568863f15a" exitCode=0 Oct 30 20:15:26.258243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.258175 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerDied","Data":"517ff5474451a592581b385f3b44b739781b900191f36818c58ce5568863f15a"} Oct 30 20:15:26.258317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.258268 4524 scope.go:122] "RemoveContainer" containerID="e2cea8dcf26cb1fa948d7853efd3e96147011ca9f3cb52b303f37ffcc13234d6" Oct 30 20:15:26.260983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.260716 4524 generic.go:356] "Generic (PLEG): container finished" podID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerID="8c1eeb668ff0a034ed1d919b4d261fe6ca58e5605169016439e8ff0075e2578b" exitCode=0 Oct 30 20:15:26.260983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.260795 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" event={"ID":"eba67f32-faaa-4efe-a13f-e68867169e7c","Type":"ContainerDied","Data":"8c1eeb668ff0a034ed1d919b4d261fe6ca58e5605169016439e8ff0075e2578b"} Oct 30 20:15:26.268536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.264077 4524 generic.go:356] "Generic (PLEG): container finished" podID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerID="9267fe4502441f6ce5e721d306b21e8cbdd4c787f9c1e47718efde87aef860a2" exitCode=0 Oct 30 20:15:26.268536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.264161 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" event={"ID":"e71fc396-ea17-426f-868d-05d5fafce8ae","Type":"ContainerDied","Data":"9267fe4502441f6ce5e721d306b21e8cbdd4c787f9c1e47718efde87aef860a2"} Oct 30 20:15:26.268536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.266596 4524 generic.go:356] "Generic (PLEG): container finished" podID="ddcf7254-2b05-412d-b788-5a48a48205da" containerID="56803420a2845ca4a772f3d3c1334e9780c56f1e3e86506e56c85bb0e1b105c8" exitCode=0 Oct 30 20:15:26.268536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.266820 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" event={"ID":"ddcf7254-2b05-412d-b788-5a48a48205da","Type":"ContainerDied","Data":"56803420a2845ca4a772f3d3c1334e9780c56f1e3e86506e56c85bb0e1b105c8"} Oct 30 20:15:26.399647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.399289 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:15:26.405492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.405418 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:15:26.411353 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.410061 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:15:26.474930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.474902 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 20:15:26.532983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.532948 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.533143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.532988 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.533143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533016 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.533273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533194 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.533273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533237 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.533386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533273 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.533386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533286 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.533386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533351 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.533535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533419 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.533535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533501 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.533647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533531 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.533647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533555 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.533647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533623 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.533647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533635 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533676 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533788 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533793 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533817 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533833 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:26.533885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533845 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533908 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533952 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533968 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.533991 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534026 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534053 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534092 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534143 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534152 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:26.534198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534177 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.534758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534266 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.534758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534422 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.534758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534727 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:26.534936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.534827 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.535149 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.535229 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.535318 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\") pod \"eba67f32-faaa-4efe-a13f-e68867169e7c\" (UID: \"eba67f32-faaa-4efe-a13f-e68867169e7c\") " Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.535352 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\") pod \"e71fc396-ea17-426f-868d-05d5fafce8ae\" (UID: \"e71fc396-ea17-426f-868d-05d5fafce8ae\") " Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.535384 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\") pod \"ddcf7254-2b05-412d-b788-5a48a48205da\" (UID: \"ddcf7254-2b05-412d-b788-5a48a48205da\") " Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536070 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536285 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536309 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536326 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536343 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536359 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcf7254-2b05-412d-b788-5a48a48205da-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536374 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536389 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536404 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536419 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba67f32-faaa-4efe-a13f-e68867169e7c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536434 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536449 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536465 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536479 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71fc396-ea17-426f-868d-05d5fafce8ae-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536479 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536544 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.536590 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537250 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537265 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537277 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537369 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537400 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "kube-api-access-5fnvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.536361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.537752 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.539217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.538240 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token" pod "ddcf7254-2b05-412d-b788-5a48a48205da" (UID: "ddcf7254-2b05-412d-b788-5a48a48205da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.539217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.538617 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:26.539217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.538624 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.539217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.539012 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx" pod "e71fc396-ea17-426f-868d-05d5fafce8ae" (UID: "e71fc396-ea17-426f-868d-05d5fafce8ae"). InnerVolumeSpecName "kube-api-access-g92wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.539217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.539119 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7" pod "eba67f32-faaa-4efe-a13f-e68867169e7c" (UID: "eba67f32-faaa-4efe-a13f-e68867169e7c"). InnerVolumeSpecName "kube-api-access-mvwg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.636916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.636890 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.636951 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637163 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637207 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637252 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637344 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637370 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637446 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.637539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637461 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.637596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637554 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637643 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:26.637762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637678 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\") pod \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\" (UID: \"41a61d0c-c691-4c79-81a5-ac5940b0c4a0\") " Oct 30 20:15:26.637931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.637911 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.638161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638132 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.638354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638337 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638356 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638368 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638377 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcf7254-2b05-412d-b788-5a48a48205da-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638385 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcf7254-2b05-412d-b788-5a48a48205da-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638396 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638408 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g92wx\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-kube-api-access-g92wx\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638417 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvwg7\" (UniqueName: \"kubernetes.io/projected/eba67f32-faaa-4efe-a13f-e68867169e7c-kube-api-access-mvwg7\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638426 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638434 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638445 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638453 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba67f32-faaa-4efe-a13f-e68867169e7c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638462 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71fc396-ea17-426f-868d-05d5fafce8ae-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638471 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638484 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71fc396-ea17-426f-868d-05d5fafce8ae-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638492 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnvq\" (UniqueName: \"kubernetes.io/projected/ddcf7254-2b05-412d-b788-5a48a48205da-kube-api-access-5fnvq\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638501 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638510 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.638875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.638519 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.639512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.639491 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.639595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.639513 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:26.639730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.639682 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "kube-api-access-pr7vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.639935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.639913 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo" pod "41a61d0c-c691-4c79-81a5-ac5940b0c4a0" (UID: "41a61d0c-c691-4c79-81a5-ac5940b0c4a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:26.740432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.740347 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.740432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.740375 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr7vb\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-kube-api-access-pr7vb\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.740432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.740390 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.740432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.740418 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a61d0c-c691-4c79-81a5-ac5940b0c4a0-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.808754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.804906 4524 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:26.808754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:26.804966 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:27.141630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.141601 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:15:27.252795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.252713 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.252918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.252842 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.252918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.252888 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.252959 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253059 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253117 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253153 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.253217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253169 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253217 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253267 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\") pod \"5c45629c-ab72-45b4-b012-1a706baecf1c\" (UID: \"5c45629c-ab72-45b4-b012-1a706baecf1c\") " Oct 30 20:15:27.253442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253371 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.253888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253524 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.253888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253613 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:27.253888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253693 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.254114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.253991 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.254114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.254011 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.254114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.254026 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.254114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.254040 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.254114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.254055 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c45629c-ab72-45b4-b012-1a706baecf1c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.256151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.256119 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "kube-api-access-xs59t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.256272 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.256228 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:27.256496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.256472 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.256799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.256778 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token" pod "5c45629c-ab72-45b4-b012-1a706baecf1c" (UID: "5c45629c-ab72-45b4-b012-1a706baecf1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.264653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.264634 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:15:27.267318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.267301 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:15:27.271846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.271817 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 20:15:27.276434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.275933 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" event={"ID":"41a61d0c-c691-4c79-81a5-ac5940b0c4a0","Type":"ContainerDied","Data":"a9213af92afdedcf954fdca956c9804bfae92f765da0f6359b587913a7a1ef33"} Oct 30 20:15:27.276434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.275994 4524 scope.go:122] "RemoveContainer" containerID="517ff5474451a592581b385f3b44b739781b900191f36818c58ce5568863f15a" Oct 30 20:15:27.276434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.276084 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk" Oct 30 20:15:27.301214 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.299494 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" Oct 30 20:15:27.309883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.309858 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" Oct 30 20:15:27.324715 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.316033 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" Oct 30 20:15:27.327336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.327310 4524 generic.go:356] "Generic (PLEG): container finished" podID="6eb20843-de34-4e51-9436-608d9d23950e" containerID="61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3" exitCode=0 Oct 30 20:15:27.327417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.327373 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" Oct 30 20:15:27.328991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.328971 4524 generic.go:356] "Generic (PLEG): container finished" podID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerID="c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a" exitCode=0 Oct 30 20:15:27.329099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.329080 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" Oct 30 20:15:27.330410 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.330390 4524 generic.go:356] "Generic (PLEG): container finished" podID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerID="825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122" exitCode=0 Oct 30 20:15:27.330505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.330472 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" Oct 30 20:15:27.332254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.332231 4524 generic.go:356] "Generic (PLEG): container finished" podID="280d856f-7580-44f5-a96e-f3fbc4355780" containerID="9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f" exitCode=0 Oct 30 20:15:27.332360 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.332347 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" Oct 30 20:15:27.354401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354375 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.354494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354407 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.354494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354445 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.354573 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354489 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.354613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354588 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.354723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354627 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.354723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354661 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.354867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354846 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.354926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354867 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.354980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.354969 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355025 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.355097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355065 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.355157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355125 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355131 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.355265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355174 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355206 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355294 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355332 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355376 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355411 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355446 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355505 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355563 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.355754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355632 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355713 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\") pod \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\" (UID: \"c453eb15-1bf0-48b9-9d46-d2c106f90caf\") " Oct 30 20:15:27.355754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355714 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.355754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355746 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.355966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355781 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.355966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355817 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355887 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\") pod \"6eb20843-de34-4e51-9436-608d9d23950e\" (UID: \"6eb20843-de34-4e51-9436-608d9d23950e\") " Oct 30 20:15:27.355966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.355920 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\") pod \"280d856f-7580-44f5-a96e-f3fbc4355780\" (UID: \"280d856f-7580-44f5-a96e-f3fbc4355780\") " Oct 30 20:15:27.356519 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356494 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.356624 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356606 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356632 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356647 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356662 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356678 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356694 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356727 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs59t\" (UniqueName: \"kubernetes.io/projected/5c45629c-ab72-45b4-b012-1a706baecf1c-kube-api-access-xs59t\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.356967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356741 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.357126 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.357073 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.357194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.357148 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.357386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.357360 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.357880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.357854 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.358090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.356757 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c45629c-ab72-45b4-b012-1a706baecf1c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.358302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.358279 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.358473 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.358368 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:27.358669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.358654 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.360594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.359451 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:27.360594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.359466 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "kube-api-access-24zzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.360594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.359811 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.360594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.360009 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.360594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.360158 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:27.361400 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.361371 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "kube-api-access-8dswt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.361754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.361712 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:27.361834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.361776 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "kube-api-access-nhmnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.362023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.362001 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.362227 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.362196 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:27.363653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.363622 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.363768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.363737 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token" pod "280d856f-7580-44f5-a96e-f3fbc4355780" (UID: "280d856f-7580-44f5-a96e-f3fbc4355780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.365093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.365071 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo" pod "6eb20843-de34-4e51-9436-608d9d23950e" (UID: "6eb20843-de34-4e51-9436-608d9d23950e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:27.366055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366032 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.366146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366052 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token" pod "c453eb15-1bf0-48b9-9d46-d2c106f90caf" (UID: "c453eb15-1bf0-48b9-9d46-d2c106f90caf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.366650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366628 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq" event={"ID":"eba67f32-faaa-4efe-a13f-e68867169e7c","Type":"ContainerDied","Data":"38fa971a43c36675b8a999a8dd602ce36e81653986cd0496ec4776ca8a82eb7f"} Oct 30 20:15:27.366731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366680 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh" event={"ID":"e71fc396-ea17-426f-868d-05d5fafce8ae","Type":"ContainerDied","Data":"e566c1cbca99df2b16f0c30e0afe0262a4717c19eccd5634af7d2e8597650e06"} Oct 30 20:15:27.366790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366730 4524 scope.go:122] "RemoveContainer" containerID="8c1eeb668ff0a034ed1d919b4d261fe6ca58e5605169016439e8ff0075e2578b" Oct 30 20:15:27.366790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366744 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64" event={"ID":"ddcf7254-2b05-412d-b788-5a48a48205da","Type":"ContainerDied","Data":"bf36fb908589563cafbe5b1d54942054ba934b4d761405ccdf5a6e7f20ea15e5"} Oct 30 20:15:27.366790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366773 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" event={"ID":"6eb20843-de34-4e51-9436-608d9d23950e","Type":"ContainerDied","Data":"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366801 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn" event={"ID":"6eb20843-de34-4e51-9436-608d9d23950e","Type":"ContainerDied","Data":"ad70d8606f4a8dd66420b6a583523c7f3f1ad6ebe0a80445417854d72e0db4da"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366824 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" event={"ID":"5c45629c-ab72-45b4-b012-1a706baecf1c","Type":"ContainerDied","Data":"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366851 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz" event={"ID":"5c45629c-ab72-45b4-b012-1a706baecf1c","Type":"ContainerDied","Data":"a8949a9f929a984d382ff6c614c5c2a94a13fed929e4ab34a2c882a14b4c5c16"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366873 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" event={"ID":"c453eb15-1bf0-48b9-9d46-d2c106f90caf","Type":"ContainerDied","Data":"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366901 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn" event={"ID":"c453eb15-1bf0-48b9-9d46-d2c106f90caf","Type":"ContainerDied","Data":"3b12dfc23ce20fc3d554b7cbf013e769624447bb9aefa4c420cf31abccfc2438"} Oct 30 20:15:27.366926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366923 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerDied","Data":"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f"} Oct 30 20:15:27.367208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.366952 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8" event={"ID":"280d856f-7580-44f5-a96e-f3fbc4355780","Type":"ContainerDied","Data":"e3951e30b6ecb5efbefd0808b3056fd28dd4a61989deb74b4c81890151016bdd"} Oct 30 20:15:27.402949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.400930 4524 scope.go:122] "RemoveContainer" containerID="9267fe4502441f6ce5e721d306b21e8cbdd4c787f9c1e47718efde87aef860a2" Oct 30 20:15:27.422852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.416074 4524 scope.go:122] "RemoveContainer" containerID="56803420a2845ca4a772f3d3c1334e9780c56f1e3e86506e56c85bb0e1b105c8" Oct 30 20:15:27.457595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.457298 4524 scope.go:122] "RemoveContainer" containerID="61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459391 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb20843-de34-4e51-9436-608d9d23950e-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459419 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459437 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459451 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459464 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459480 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459494 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459510 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459525 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dswt\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-kube-api-access-8dswt\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459539 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459552 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459566 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb20843-de34-4e51-9436-608d9d23950e-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459579 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459593 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280d856f-7580-44f5-a96e-f3fbc4355780-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459607 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280d856f-7580-44f5-a96e-f3fbc4355780-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459621 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459634 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c453eb15-1bf0-48b9-9d46-d2c106f90caf-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459648 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459662 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhmnz\" (UniqueName: \"kubernetes.io/projected/6eb20843-de34-4e51-9436-608d9d23950e-kube-api-access-nhmnz\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459677 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zzj\" (UniqueName: \"kubernetes.io/projected/280d856f-7580-44f5-a96e-f3fbc4355780-kube-api-access-24zzj\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459691 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280d856f-7580-44f5-a96e-f3fbc4355780-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.460899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.459723 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb20843-de34-4e51-9436-608d9d23950e-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.480647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.480435 4524 scope.go:122] "RemoveContainer" containerID="61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3" Oct 30 20:15:27.480897 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:27.480843 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3\": container with ID starting with 61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3 not found: ID does not exist" containerID="61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3" Oct 30 20:15:27.481229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.480993 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3"} err="failed to get container status \"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3\": rpc error: code = NotFound desc = could not find container \"61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3\": container with ID starting with 61a58aacb5497b57e90235c35d6f3e4d8a2fb5f2f35d950ac9822de0f9d8e6f3 not found: ID does not exist" Oct 30 20:15:27.481229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.481019 4524 scope.go:122] "RemoveContainer" containerID="c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a" Oct 30 20:15:27.498498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.498471 4524 scope.go:122] "RemoveContainer" containerID="c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a" Oct 30 20:15:27.499269 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:27.498898 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a\": container with ID starting with c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a not found: ID does not exist" containerID="c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a" Oct 30 20:15:27.499269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.498943 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a"} err="failed to get container status \"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a\": rpc error: code = NotFound desc = could not find container \"c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a\": container with ID starting with c64db68627e2fc0795515775278a186531b7c368c0eed9de05db5dca20af2c5a not found: ID does not exist" Oct 30 20:15:27.499269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.498968 4524 scope.go:122] "RemoveContainer" containerID="825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122" Oct 30 20:15:27.526532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.526505 4524 scope.go:122] "RemoveContainer" containerID="825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122" Oct 30 20:15:27.528965 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:27.527222 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122\": container with ID starting with 825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122 not found: ID does not exist" containerID="825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122" Oct 30 20:15:27.528965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.527265 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122"} err="failed to get container status \"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122\": rpc error: code = NotFound desc = could not find container \"825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122\": container with ID starting with 825b2051f504572bc2e0bfcb5e521129e6f0d7ea7edb5ad1430dbb987151a122 not found: ID does not exist" Oct 30 20:15:27.528965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.527292 4524 scope.go:122] "RemoveContainer" containerID="9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f" Oct 30 20:15:27.540123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.540102 4524 scope.go:122] "RemoveContainer" containerID="66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3" Oct 30 20:15:27.551956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.551937 4524 scope.go:122] "RemoveContainer" containerID="9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f" Oct 30 20:15:27.552310 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:27.552283 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f\": container with ID starting with 9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f not found: ID does not exist" containerID="9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f" Oct 30 20:15:27.552434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.552323 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f"} err="failed to get container status \"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f\": rpc error: code = NotFound desc = could not find container \"9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f\": container with ID starting with 9ad793b9e7a6021b5993b948d705ade039442347b8e5e0da6f2703ef8d143a3f not found: ID does not exist" Oct 30 20:15:27.552434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.552348 4524 scope.go:122] "RemoveContainer" containerID="66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3" Oct 30 20:15:27.552676 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:27.552648 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3\": container with ID starting with 66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3 not found: ID does not exist" containerID="66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3" Oct 30 20:15:27.552830 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.552684 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3"} err="failed to get container status \"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3\": rpc error: code = NotFound desc = could not find container \"66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3\": container with ID starting with 66cf58a1abe22850a2900f674e964be0d6f4c912be92f8f92ad3a7b094ee57e3 not found: ID does not exist" Oct 30 20:15:27.588396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.588373 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:15:27.694784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.694756 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:15:27.699669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.699639 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-kjk64"] Oct 30 20:15:27.764163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764131 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764175 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764278 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764323 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764391 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764528 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764548 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:27.764593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764557 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764584 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764638 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\") pod \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\" (UID: \"71c844a2-dfd9-45bf-84c5-0b27690a0eb9\") " Oct 30 20:15:27.764819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764655 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.764819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764667 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.764976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764919 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.764976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.764942 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.765398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.765373 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.765626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.765593 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.765626 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.765617 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.765815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.765635 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.765815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.765651 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.767047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.767025 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.767047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.767037 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "kube-api-access-pf6tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.767229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.767211 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:27.767324 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.767302 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy" pod "71c844a2-dfd9-45bf-84c5-0b27690a0eb9" (UID: "71c844a2-dfd9-45bf-84c5-0b27690a0eb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.866752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.866657 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6tc\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-kube-api-access-pf6tc\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.866752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.866686 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.866752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.866711 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.866752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.866726 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c844a2-dfd9-45bf-84c5-0b27690a0eb9-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.867171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.867148 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:15:27.871738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.871692 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-zbbfz"] Oct 30 20:15:27.984531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.984497 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:15:27.991513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:27.991487 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-f5kxh"] Oct 30 20:15:28.227413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.227376 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:15:28.232300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.232272 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4n2qq"] Oct 30 20:15:28.345469 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.345432 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 20:15:28.348918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.348883 4524 generic.go:356] "Generic (PLEG): container finished" podID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerID="e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2" exitCode=0 Oct 30 20:15:28.349131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.349097 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" Oct 30 20:15:28.349727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.349684 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" event={"ID":"71c844a2-dfd9-45bf-84c5-0b27690a0eb9","Type":"ContainerDied","Data":"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2"} Oct 30 20:15:28.349915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.349888 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz" event={"ID":"71c844a2-dfd9-45bf-84c5-0b27690a0eb9","Type":"ContainerDied","Data":"80a6894e64d2fa02a879583861a1450d8ba40905b72043582f6120eee5767bb4"} Oct 30 20:15:28.350028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.349933 4524 scope.go:122] "RemoveContainer" containerID="e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2" Oct 30 20:15:28.351196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.350957 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-tzrgk"] Oct 30 20:15:28.365518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.365492 4524 scope.go:122] "RemoveContainer" containerID="e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2" Oct 30 20:15:28.366620 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:28.366595 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2\": container with ID starting with e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2 not found: ID does not exist" containerID="e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2" Oct 30 20:15:28.366744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.366632 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2"} err="failed to get container status \"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2\": rpc error: code = NotFound desc = could not find container \"e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2\": container with ID starting with e5a3f8e6778afdfb9453e0b3daca0c8d8ee2a25df9768fb904e7ddf4344e2cc2 not found: ID does not exist" Oct 30 20:15:28.606773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.600096 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:15:28.606773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.605532 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-f5dwn"] Oct 30 20:15:28.706814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.706784 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 20:15:28.716072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.716046 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-msdk8"] Oct 30 20:15:28.827523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.827486 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:15:28.835455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.835429 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-j7lvn"] Oct 30 20:15:28.979221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.979184 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:15:28.992869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:28.992840 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wd5nz"] Oct 30 20:15:29.239363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.239290 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:15:29.303612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.303581 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" path="/var/lib/kubelet/pods/280d856f-7580-44f5-a96e-f3fbc4355780/volumes" Oct 30 20:15:29.304078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.304062 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" path="/var/lib/kubelet/pods/41a61d0c-c691-4c79-81a5-ac5940b0c4a0/volumes" Oct 30 20:15:29.304514 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.304501 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" path="/var/lib/kubelet/pods/5c45629c-ab72-45b4-b012-1a706baecf1c/volumes" Oct 30 20:15:29.305189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.305172 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb20843-de34-4e51-9436-608d9d23950e" path="/var/lib/kubelet/pods/6eb20843-de34-4e51-9436-608d9d23950e/volumes" Oct 30 20:15:29.305612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.305599 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" path="/var/lib/kubelet/pods/71c844a2-dfd9-45bf-84c5-0b27690a0eb9/volumes" Oct 30 20:15:29.306059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.306046 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" path="/var/lib/kubelet/pods/c453eb15-1bf0-48b9-9d46-d2c106f90caf/volumes" Oct 30 20:15:29.306851 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.306835 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" path="/var/lib/kubelet/pods/ddcf7254-2b05-412d-b788-5a48a48205da/volumes" Oct 30 20:15:29.307253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.307240 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" path="/var/lib/kubelet/pods/e71fc396-ea17-426f-868d-05d5fafce8ae/volumes" Oct 30 20:15:29.307935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:29.307921 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" path="/var/lib/kubelet/pods/eba67f32-faaa-4efe-a13f-e68867169e7c/volumes" Oct 30 20:15:30.909781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:30.905692 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:35.828722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:35.826886 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.925751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:39.924257 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.484974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.484941 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:15:40.485561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485548 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485563 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485576 4524 cpu_manager.go:414] "containerMap: removing container" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485581 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485593 4524 cpu_manager.go:414] "containerMap: removing container" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485598 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485608 4524 cpu_manager.go:414] "containerMap: removing container" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" Oct 30 20:15:40.485617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485613 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485621 4524 cpu_manager.go:414] "containerMap: removing container" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485626 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485663 4524 cpu_manager.go:414] "containerMap: removing container" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485668 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485675 4524 cpu_manager.go:414] "containerMap: removing container" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485680 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485688 4524 cpu_manager.go:414] "containerMap: removing container" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485693 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485715 4524 cpu_manager.go:414] "containerMap: removing container" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485719 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485726 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485730 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485739 4524 cpu_manager.go:414] "containerMap: removing container" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485743 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485836 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485844 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba67f32-faaa-4efe-a13f-e68867169e7c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485852 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c45629c-ab72-45b4-b012-1a706baecf1c" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485858 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcf7254-2b05-412d-b788-5a48a48205da" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485866 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a61d0c-c691-4c79-81a5-ac5940b0c4a0" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485874 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71fc396-ea17-426f-868d-05d5fafce8ae" containerName="istio-proxy" Oct 30 20:15:40.485874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485881 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb20843-de34-4e51-9436-608d9d23950e" containerName="istio-proxy" Oct 30 20:15:40.486532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485888 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.486532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485896 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c844a2-dfd9-45bf-84c5-0b27690a0eb9" containerName="istio-proxy" Oct 30 20:15:40.486532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485902 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="280d856f-7580-44f5-a96e-f3fbc4355780" containerName="istio-proxy" Oct 30 20:15:40.486532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.485910 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="c453eb15-1bf0-48b9-9d46-d2c106f90caf" containerName="istio-proxy" Oct 30 20:15:40.489419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.489403 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.543515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.543484 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-77fcfa-serving-cert\"" Oct 30 20:15:40.547668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.547630 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnvb\" (UniqueName: \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.547809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.547693 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.547809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.547776 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.548026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.548003 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.597250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.597216 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:15:40.648818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.648785 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnvb\" (UniqueName: \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.649009 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.648843 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.649009 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.648883 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.649150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.649133 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.649285 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:40.649264 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-serving-cert: secret "maas-api-e2e-api-a-77fcfa-serving-cert" not found Oct 30 20:15:40.649353 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:40.649341 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls podName:44c2b553-1ba8-49cc-b53f-c73ff3d3fad9 nodeName:}" failed. No retries permitted until 2026-10-30 20:15:41.149318638 +0000 UTC m=+2035.992990248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls") pod "maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" (UID: "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9") : secret "maas-api-e2e-api-a-77fcfa-serving-cert" not found Oct 30 20:15:40.649670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.649647 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.651524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.651502 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:40.719335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:40.716583 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnvb\" (UniqueName: \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:41.176422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:41.176215 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:41.185514 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:41.185485 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") pod \"maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:41.404264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:41.404230 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:41.790226 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:41.790199 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:15:41.791570 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:15:41.791537 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c2b553_1ba8_49cc_b53f_c73ff3d3fad9.slice/crio-7020d2fde8039ada9f5e70febb751977dc553021656c1ccc2f1e65bd5b0138fa WatchSource:0}: Error finding container 7020d2fde8039ada9f5e70febb751977dc553021656c1ccc2f1e65bd5b0138fa: Status 404 returned error can't find the container with id 7020d2fde8039ada9f5e70febb751977dc553021656c1ccc2f1e65bd5b0138fa Oct 30 20:15:41.936461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:41.936428 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.287004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:42.286954 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.472294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:42.472259 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" event={"ID":"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9","Type":"ContainerStarted","Data":"7020d2fde8039ada9f5e70febb751977dc553021656c1ccc2f1e65bd5b0138fa"} Oct 30 20:15:42.790693 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:15:42.790655 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:15:43.481058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:43.481006 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" event={"ID":"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9","Type":"ContainerStarted","Data":"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a"} Oct 30 20:15:43.481563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:43.481103 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:43.606064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:43.606014 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" podStartSLOduration=2.7821298949999997 podStartE2EDuration="3.60600208s" totalImagesPullingTime="823.872185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:43.602782907 +0000 UTC m=+2038.446454533" watchObservedRunningTime="2026-10-30 20:15:43.60600208 +0000 UTC m=+2038.449673708" Oct 30 20:15:45.908332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:45.908304 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:46.267583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:46.267544 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.426492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:48.426448 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:49.495942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:49.495913 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:15:51.316630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:51.314442 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.667916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:52.667870 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.277873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:53.277838 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.375236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.375201 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:15:54.381486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.381463 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416316 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84z94\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416368 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416439 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416477 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416524 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416586 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416608 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416664 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.416818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.416801 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.436080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.436032 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-dc56e1-openshift-default-dockercfg-xnkct\"" Oct 30 20:15:54.493001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.492967 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:15:54.524616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.524579 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.524989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.524960 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.553260 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.553228 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84z94\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.554001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.553942 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.557820 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.557783 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.629515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.629431 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.735378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.552039 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.735378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.553015 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.735668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.527268 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.735668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.631472 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.735668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.735604 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.754185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.735949 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.754185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.735989 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.754185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.736735 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.754185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.737154 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.754185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.737473 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.810721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.790005 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.828641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.828608 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84z94\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.829263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.829239 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\") pod \"e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:54.971598 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.971545 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.996956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:54.996925 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:55.473733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.471368 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:15:55.479046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.479006 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:55.479143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.479092 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:55.479143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.479133 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:55.687220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.687179 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerStarted","Data":"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9"} Oct 30 20:15:55.687428 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.687239 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerStarted","Data":"bbb4f8329107380d3d036454df9ac9ba8e9da4bd3773116756679375d0b0a2aa"} Oct 30 20:15:55.809927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.809828 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podStartSLOduration=1.809811962 podStartE2EDuration="1.809811962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:55.807399451 +0000 UTC m=+2050.651071081" watchObservedRunningTime="2026-10-30 20:15:55.809811962 +0000 UTC m=+2050.653483592" Oct 30 20:15:55.998904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:55.998810 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:57.003396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:57.003364 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:57.078725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:57.069576 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.701770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:57.701739 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:57.702618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:57.702596 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:15:58.729029 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:58.728993 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.056306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:15:59.056215 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.401002 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.400965 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.603220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.603184 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:16:02.608509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.608491 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.640291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.640259 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.662213 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.662161 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-dc56e1\"" Oct 30 20:16:02.712685 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.712643 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.712926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.712879 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmhz\" (UniqueName: \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.715537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.715512 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:16:02.718941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.718903 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:16:02.730435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.730408 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.815723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.814858 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.815723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.814990 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.815723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.815079 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5x9k\" (UniqueName: \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.815723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.815223 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmhz\" (UniqueName: \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.832049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.816564 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.833625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.833594 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-dc56e1-dockercfg-jgl72\"" Oct 30 20:16:02.920766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.920124 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.920766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.920266 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5x9k\" (UniqueName: \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.921072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.921050 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:02.922796 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.922770 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmhz\" (UniqueName: \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\") pod \"payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.922995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.922979 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:02.947192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.947166 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:16:02.987327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:02.987293 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5x9k\" (UniqueName: \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\") pod \"payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:03.165164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.165128 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:03.362996 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.362965 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:16:03.363951 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:03.363926 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bce1e32_198f_40d5_801d_542c4dae7282.slice/crio-b3d3b6a71a9bfad9c9a90d997f77275491372c8e05dab720c2b051abee39afde WatchSource:0}: Error finding container b3d3b6a71a9bfad9c9a90d997f77275491372c8e05dab720c2b051abee39afde: Status 404 returned error can't find the container with id b3d3b6a71a9bfad9c9a90d997f77275491372c8e05dab720c2b051abee39afde Oct 30 20:16:03.549388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.549360 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:16:03.550409 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:03.550382 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cea4c52_1a5c_4457_b387_b5dd65357a91.slice/crio-ba5807c4a30716d28310e6c1efeae72d236538a6083a2d368c827994c2e4f682 WatchSource:0}: Error finding container ba5807c4a30716d28310e6c1efeae72d236538a6083a2d368c827994c2e4f682: Status 404 returned error can't find the container with id ba5807c4a30716d28310e6c1efeae72d236538a6083a2d368c827994c2e4f682 Oct 30 20:16:03.803037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.802999 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" event={"ID":"0bce1e32-198f-40d5-801d-542c4dae7282","Type":"ContainerStarted","Data":"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3"} Oct 30 20:16:03.803736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.803071 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:03.803736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.803096 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" event={"ID":"0bce1e32-198f-40d5-801d-542c4dae7282","Type":"ContainerStarted","Data":"b3d3b6a71a9bfad9c9a90d997f77275491372c8e05dab720c2b051abee39afde"} Oct 30 20:16:03.804834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.804808 4524 generic.go:356] "Generic (PLEG): container finished" podID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerID="f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a" exitCode=0 Oct 30 20:16:03.804965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.804884 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerDied","Data":"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a"} Oct 30 20:16:03.805325 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.805302 4524 scope.go:122] "RemoveContainer" containerID="f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a" Oct 30 20:16:03.806653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.806628 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" event={"ID":"0cea4c52-1a5c-4457-b387-b5dd65357a91","Type":"ContainerStarted","Data":"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208"} Oct 30 20:16:03.806770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.806673 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" event={"ID":"0cea4c52-1a5c-4457-b387-b5dd65357a91","Type":"ContainerStarted","Data":"ba5807c4a30716d28310e6c1efeae72d236538a6083a2d368c827994c2e4f682"} Oct 30 20:16:03.806770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.806740 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:03.807056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.807029 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:03.807112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.807097 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:03.807165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.807144 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:03.924102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:03.924053 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" podStartSLOduration=1.924037008 podStartE2EDuration="1.924037008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:03.921449709 +0000 UTC m=+2058.765121336" watchObservedRunningTime="2026-10-30 20:16:03.924037008 +0000 UTC m=+2058.767708637" Oct 30 20:16:04.815099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:04.815059 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerStarted","Data":"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0"} Oct 30 20:16:04.941100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:04.941043 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" podStartSLOduration=2.941025833 podStartE2EDuration="2.941025833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:04.180412619 +0000 UTC m=+2059.024084284" watchObservedRunningTime="2026-10-30 20:16:04.941025833 +0000 UTC m=+2059.784697467" Oct 30 20:16:05.222334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:05.222299 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:16:05.222544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:05.222520 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 20:16:05.222613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:05.222590 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 20:16:06.222644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:06.222607 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 20:16:06.223062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:06.222670 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 20:16:07.222283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:07.222248 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 20:16:07.222492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:07.222312 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 20:16:07.669320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:07.669284 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:08.222432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:08.222396 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 20:16:08.222616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:08.222471 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 20:16:09.222119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:09.222088 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 20:16:09.222493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:09.222147 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 20:16:10.227475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:10.227443 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:16:10.227930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:10.227916 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:16:10.229104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:10.229084 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:16:12.536535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:12.536497 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.787632 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:16:12.787539 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:16:14.364411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:14.364375 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.582486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:14.582447 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.818294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:14.818265 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:16:14.818476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:14.818375 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:16:16.769980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:16.769937 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.427222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:18.427185 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.676219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:18.676180 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.897501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:18.897465 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:20.492281 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:20.492243 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:16:20.493142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:20.493088 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" containerID="cri-o://6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a" gracePeriod=30 Oct 30 20:16:20.527064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:20.527038 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:23.537597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.537566 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:16:23.542030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.542013 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.596380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.596357 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5d2c70-serving-cert\"" Oct 30 20:16:23.649900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.649874 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:16:23.694115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.694064 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.694302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.694276 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.694429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.694410 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkvj8\" (UniqueName: \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.694520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.694499 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.795809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.795733 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkvj8\" (UniqueName: \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.795952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.795813 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.795952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.795888 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.796034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.795979 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.796563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.796539 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.798361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.798333 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.798450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.798341 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:23.854120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.854085 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:23.856639 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:23.856615 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkvj8\" (UniqueName: \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\") pod \"maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:24.065847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:24.065770 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:24.155648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:24.155617 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:24.536610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:24.536582 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:16:24.538637 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:24.538609 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6338bd1_4f84_4849_bea8_f8c04e6966ea.slice/crio-9a28aac72e6ee05641546ee23943da52f832427138a42cf547e6ffbe5b7daa34 WatchSource:0}: Error finding container 9a28aac72e6ee05641546ee23943da52f832427138a42cf547e6ffbe5b7daa34: Status 404 returned error can't find the container with id 9a28aac72e6ee05641546ee23943da52f832427138a42cf547e6ffbe5b7daa34 Oct 30 20:16:24.955323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:24.955290 4524 patch_prober.go:29] interesting pod/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:16:24.955529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:24.955349 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:16:25.007706 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.007658 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" event={"ID":"f6338bd1-4f84-4849-bea8-f8c04e6966ea","Type":"ContainerStarted","Data":"9a28aac72e6ee05641546ee23943da52f832427138a42cf547e6ffbe5b7daa34"} Oct 30 20:16:25.607067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.607034 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:25.800786 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.800761 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:16:25.854244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854215 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854294 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854364 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854419 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854474 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854573 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854615 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854670 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.854826 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854762 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:25.854888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854844 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\") pod \"476fa658-bc72-41c9-8ad7-b718b4f3280f\" (UID: \"476fa658-bc72-41c9-8ad7-b718b4f3280f\") " Oct 30 20:16:25.855019 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.854995 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:25.855157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855136 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:25.855341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855297 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:25.855465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855445 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:25.855587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855565 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.855657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855594 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.855657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855625 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476fa658-bc72-41c9-8ad7-b718b4f3280f-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.855657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855641 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.855657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.855656 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.856774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.856728 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:25.856879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.856795 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "kube-api-access-k996m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:25.857266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.857249 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:25.857309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.857261 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy" pod "476fa658-bc72-41c9-8ad7-b718b4f3280f" (UID: "476fa658-bc72-41c9-8ad7-b718b4f3280f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:25.958746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.958294 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.958746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.958335 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k996m\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-kube-api-access-k996m\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.958746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.958351 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.958746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:25.958364 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476fa658-bc72-41c9-8ad7-b718b4f3280f-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.016403 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.016362 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" event={"ID":"f6338bd1-4f84-4849-bea8-f8c04e6966ea","Type":"ContainerStarted","Data":"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950"} Oct 30 20:16:26.016592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.016430 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:26.018113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.018088 4524 generic.go:356] "Generic (PLEG): container finished" podID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerID="6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a" exitCode=0 Oct 30 20:16:26.018246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.018137 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" Oct 30 20:16:26.018246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.018166 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerDied","Data":"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a"} Oct 30 20:16:26.018246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.018216 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q" event={"ID":"476fa658-bc72-41c9-8ad7-b718b4f3280f","Type":"ContainerDied","Data":"472972b62aaef0df8af76fdc6f01141d5685d77a217322851d49487115281c1f"} Oct 30 20:16:26.018353 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.018246 4524 scope.go:122] "RemoveContainer" containerID="6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a" Oct 30 20:16:26.029515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.029489 4524 scope.go:122] "RemoveContainer" containerID="ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af" Oct 30 20:16:26.038689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.038670 4524 scope.go:122] "RemoveContainer" containerID="6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a" Oct 30 20:16:26.039027 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:16:26.039008 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a\": container with ID starting with 6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a not found: ID does not exist" containerID="6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a" Oct 30 20:16:26.039103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.039034 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a"} err="failed to get container status \"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a\": rpc error: code = NotFound desc = could not find container \"6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a\": container with ID starting with 6124aa7b6a348084433be8a87e6687e191384be5d511522e4a129df7e0d2dd8a not found: ID does not exist" Oct 30 20:16:26.039103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.039050 4524 scope.go:122] "RemoveContainer" containerID="ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af" Oct 30 20:16:26.039319 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:16:26.039301 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af\": container with ID starting with ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af not found: ID does not exist" containerID="ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af" Oct 30 20:16:26.039368 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.039327 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af"} err="failed to get container status \"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af\": rpc error: code = NotFound desc = could not find container \"ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af\": container with ID starting with ef64b82a970f4d8668c5b56ea0f4b647fdfe115dde35318fb99bfc8cd33a42af not found: ID does not exist" Oct 30 20:16:26.282445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.282353 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:16:26.310725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.310329 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-577fb85a-openshift-default-5fc9575cbc-tnm5q"] Oct 30 20:16:26.316026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.315984 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:16:26.318315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.317307 4524 cpu_manager.go:414] "containerMap: removing container" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:26.318510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.318489 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:26.318982 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.318917 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:26.319104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.319093 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:26.325885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.325862 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.382886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.382855 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-25cc2f-openshift-default-dockercfg-wl58l\"" Oct 30 20:16:26.386387 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.386354 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:16:26.401744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.400153 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" podStartSLOduration=2.571333287 podStartE2EDuration="3.400123148s" totalImagesPullingTime="828.789861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.39697379 +0000 UTC m=+2081.240645421" watchObservedRunningTime="2026-10-30 20:16:26.400123148 +0000 UTC m=+2081.243794778" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511515 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511578 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511634 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511736 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511766 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511794 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511821 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511872 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.512861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.511906 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lxp\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.544450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.544268 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.617738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.613992 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.637967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.637931 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638275 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638456 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638564 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638685 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638672 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638808 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.638962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.638949 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.639076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.639060 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lxp\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.645936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.643964 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.645936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.644152 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.645936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.644432 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.645936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.644653 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.645936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.644741 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.649637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.649590 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.680076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.680042 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.702317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.702290 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lxp\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.706204 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.706180 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\") pod \"e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:26.821438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.821345 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.961663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:26.961622 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:27.301448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:27.301413 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" path="/var/lib/kubelet/pods/476fa658-bc72-41c9-8ad7-b718b4f3280f/volumes" Oct 30 20:16:27.437723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:27.428664 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:16:27.452719 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:27.442226 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844d0230_0706_4c0e_a80c_1133fdc14d35.slice/crio-abb6a3fc23560ba54c815470b3d31b33e3246e61d13afcf172dc2f14fe4c9ba6 WatchSource:0}: Error finding container abb6a3fc23560ba54c815470b3d31b33e3246e61d13afcf172dc2f14fe4c9ba6: Status 404 returned error can't find the container with id abb6a3fc23560ba54c815470b3d31b33e3246e61d13afcf172dc2f14fe4c9ba6 Oct 30 20:16:27.478716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:27.474432 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:27.478716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:27.474530 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:27.478716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:27.474575 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:28.039018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:28.038977 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerStarted","Data":"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7"} Oct 30 20:16:28.039482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:28.039036 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerStarted","Data":"abb6a3fc23560ba54c815470b3d31b33e3246e61d13afcf172dc2f14fe4c9ba6"} Oct 30 20:16:28.262417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:28.262358 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podStartSLOduration=2.262344735 podStartE2EDuration="2.262344735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:28.209782916 +0000 UTC m=+2083.053454545" watchObservedRunningTime="2026-10-30 20:16:28.262344735 +0000 UTC m=+2083.106016363" Oct 30 20:16:28.961880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:28.961850 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:28.967918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:28.967891 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:29.045041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:29.045012 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:29.045911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:29.045883 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:16:29.468746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:29.468681 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:31.189254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:31.189219 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:31.415196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:31.415150 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:31.641490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:31.641460 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.037447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:32.037374 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:16:34.643534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:34.643494 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:34.875466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:34.875400 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.000612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.000532 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:16:35.001904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.001882 4524 cpu_manager.go:414] "containerMap: removing container" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:35.002053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.002041 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476fa658-bc72-41c9-8ad7-b718b4f3280f" containerName="istio-proxy" Oct 30 20:16:35.008028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.008004 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.045401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.045370 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.045565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.045549 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lfw\" (UniqueName: \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.063139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.063106 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:16:35.063486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.063456 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-25cc2f-dockercfg-zl48p\"" Oct 30 20:16:35.063635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.063617 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-25cc2f\"" Oct 30 20:16:35.220759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.186809 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lfw\" (UniqueName: \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.220759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.187049 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.220759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.188288 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.287775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.272829 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:16:35.358879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.358838 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lfw\" (UniqueName: \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\") pod \"payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.363063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.363037 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.378435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.378398 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:16:35.390621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.390590 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jtl\" (UniqueName: \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.390787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.390734 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.492289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.492258 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jtl\" (UniqueName: \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.492445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.492319 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.493047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.493025 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.554670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.554596 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jtl\" (UniqueName: \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\") pod \"payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:35.631161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.631129 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:35.689657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:35.689327 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:36.014060 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:36.014016 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadebf0f7_b53d_42de_aa0a_41ce8a8b95b2.slice/crio-39236287cea6de9e2b8531f9f5c091ec57e1c73ddd829ae930dbadd682615201 WatchSource:0}: Error finding container 39236287cea6de9e2b8531f9f5c091ec57e1c73ddd829ae930dbadd682615201: Status 404 returned error can't find the container with id 39236287cea6de9e2b8531f9f5c091ec57e1c73ddd829ae930dbadd682615201 Oct 30 20:16:36.014526 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.014502 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:16:36.115469 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.115439 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:16:36.116255 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:16:36.116230 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02608af_9cf6_4723_a99e_0fb13d333870.slice/crio-e68050dfd2158613c07c26ff70c84c8c1f96c2eea397100643f78e67d2925445 WatchSource:0}: Error finding container e68050dfd2158613c07c26ff70c84c8c1f96c2eea397100643f78e67d2925445: Status 404 returned error can't find the container with id e68050dfd2158613c07c26ff70c84c8c1f96c2eea397100643f78e67d2925445 Oct 30 20:16:36.317258 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.317182 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" event={"ID":"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2","Type":"ContainerStarted","Data":"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff"} Oct 30 20:16:36.317421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.317390 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:36.317503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.317437 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" event={"ID":"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2","Type":"ContainerStarted","Data":"39236287cea6de9e2b8531f9f5c091ec57e1c73ddd829ae930dbadd682615201"} Oct 30 20:16:36.319025 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.319000 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" event={"ID":"b02608af-9cf6-4723-a99e-0fb13d333870","Type":"ContainerStarted","Data":"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081"} Oct 30 20:16:36.319145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.319039 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" event={"ID":"b02608af-9cf6-4723-a99e-0fb13d333870","Type":"ContainerStarted","Data":"e68050dfd2158613c07c26ff70c84c8c1f96c2eea397100643f78e67d2925445"} Oct 30 20:16:36.319210 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.319184 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:36.436571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.436508 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" podStartSLOduration=2.4364894440000002 podStartE2EDuration="2.436489444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:36.435676669 +0000 UTC m=+2091.279348298" watchObservedRunningTime="2026-10-30 20:16:36.436489444 +0000 UTC m=+2091.280161075" Oct 30 20:16:36.591534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:36.574482 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" podStartSLOduration=1.574454394 podStartE2EDuration="1.574454394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:36.553288455 +0000 UTC m=+2091.396960102" watchObservedRunningTime="2026-10-30 20:16:36.574454394 +0000 UTC m=+2091.418126025" Oct 30 20:16:39.846750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:39.846712 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:42.787304 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:16:42.787270 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:16:44.653197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:44.653155 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:46.253926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:46.253846 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.329361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:47.329330 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:16:47.329867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:47.329407 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:16:49.450751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:49.450692 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.639855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:54.639815 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:59.839617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:16:59.839397 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:04.739522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:04.739489 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.203903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:09.203867 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:17:09.578839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:09.578760 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.785927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:09.785884 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.401939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:11.401903 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.629488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:11.629455 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:12.787642 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:12.787607 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:17:14.204872 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.204828 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" podUID="cf4372de-29fa-4564-b59c-6c6ff834d752" containerName="payload-processing" containerID="cri-o://aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a" gracePeriod=25 Oct 30 20:17:14.490544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.490517 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:17:14.518777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.518744 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\" (UniqueName: \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\") pod \"cf4372de-29fa-4564-b59c-6c6ff834d752\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " Oct 30 20:17:14.518919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.518870 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\") pod \"cf4372de-29fa-4564-b59c-6c6ff834d752\" (UID: \"cf4372de-29fa-4564-b59c-6c6ff834d752\") " Oct 30 20:17:14.519228 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.519206 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume" pod "cf4372de-29fa-4564-b59c-6c6ff834d752" (UID: "cf4372de-29fa-4564-b59c-6c6ff834d752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:14.521011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.520990 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh" pod "cf4372de-29fa-4564-b59c-6c6ff834d752" (UID: "cf4372de-29fa-4564-b59c-6c6ff834d752"). InnerVolumeSpecName "kube-api-access-pj8zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:14.598265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.598226 4524 generic.go:356] "Generic (PLEG): container finished" podID="cf4372de-29fa-4564-b59c-6c6ff834d752" containerID="aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a" exitCode=0 Oct 30 20:17:14.598397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.598273 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" event={"ID":"cf4372de-29fa-4564-b59c-6c6ff834d752","Type":"ContainerDied","Data":"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a"} Oct 30 20:17:14.598397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.598277 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" Oct 30 20:17:14.598397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.598324 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn" event={"ID":"cf4372de-29fa-4564-b59c-6c6ff834d752","Type":"ContainerDied","Data":"f6c67a81bb3f2e4726c59a57a2546b41b6268bd9e26962bea5be75bbaea442c1"} Oct 30 20:17:14.598397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.598349 4524 scope.go:122] "RemoveContainer" containerID="aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a" Oct 30 20:17:14.620635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.620578 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj8zh\" (UniqueName: \"kubernetes.io/projected/cf4372de-29fa-4564-b59c-6c6ff834d752-kube-api-access-pj8zh\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:14.620635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.620613 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4372de-29fa-4564-b59c-6c6ff834d752-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:14.631204 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.631093 4524 scope.go:122] "RemoveContainer" containerID="aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a" Oct 30 20:17:14.633023 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:14.632938 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a\": container with ID starting with aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a not found: ID does not exist" containerID="aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a" Oct 30 20:17:14.633023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.632978 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a"} err="failed to get container status \"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a\": rpc error: code = NotFound desc = could not find container \"aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a\": container with ID starting with aa996aee56cae415f54c4dd6a932603e4d02e692b8b6636a07a053897442021a not found: ID does not exist" Oct 30 20:17:14.769201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.769081 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.838355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.838316 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:17:14.854284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:14.854254 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7961ed-66cdcc7d94-kv8kn"] Oct 30 20:17:15.303770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:15.303737 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4372de-29fa-4564-b59c-6c6ff834d752" path="/var/lib/kubelet/pods/cf4372de-29fa-4564-b59c-6c6ff834d752/volumes" Oct 30 20:17:20.107819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:20.104387 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:23.848239 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:23.848196 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:17:24.309224 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.309181 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.520689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.520656 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.807356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.807325 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:17:24.808010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.807996 4524 cpu_manager.go:414] "containerMap: removing container" podUID="cf4372de-29fa-4564-b59c-6c6ff834d752" containerName="payload-processing" Oct 30 20:17:24.808061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.808011 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4372de-29fa-4564-b59c-6c6ff834d752" containerName="payload-processing" Oct 30 20:17:24.808142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.808133 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4372de-29fa-4564-b59c-6c6ff834d752" containerName="payload-processing" Oct 30 20:17:24.813310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.813289 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:24.867805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.867778 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-dc56e1-serving-cert\"" Oct 30 20:17:24.873082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.873055 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:17:24.901873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.901832 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:24.901873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.901867 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:24.902093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.901996 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rpj\" (UniqueName: \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:24.902093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:24.902048 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.003493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.003444 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.003493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.003494 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.003779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.003595 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rpj\" (UniqueName: \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.003779 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:25.003623 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-serving-cert: secret "maas-api-e2e-shared-b-w4-dc56e1-serving-cert" not found Oct 30 20:17:25.003779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.003674 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.003779 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:25.003732 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls podName:ac7bd72a-66dc-489c-b646-ea06c987b9f8 nodeName:}" failed. No retries permitted until 2026-10-30 20:17:25.503692177 +0000 UTC m=+2140.347363797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls") pod "maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" (UID: "ac7bd72a-66dc-489c-b646-ea06c987b9f8") : secret "maas-api-e2e-shared-b-w4-dc56e1-serving-cert" not found Oct 30 20:17:25.004684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.004663 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.006923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.006897 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.101998 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.101902 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.111030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.110992 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rpj\" (UniqueName: \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.345726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.340347 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.527728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.516973 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.547887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.545403 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:25.731415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:25.730077 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:26.216186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.216152 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:17:26.216662 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:17:26.216639 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7bd72a_66dc_489c_b646_ea06c987b9f8.slice/crio-24a60a7a324ec7d1694933efe467ea7d718d6507ed54509198aed10adf51cf7b WatchSource:0}: Error finding container 24a60a7a324ec7d1694933efe467ea7d718d6507ed54509198aed10adf51cf7b: Status 404 returned error can't find the container with id 24a60a7a324ec7d1694933efe467ea7d718d6507ed54509198aed10adf51cf7b Oct 30 20:17:26.392427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.392386 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:17:26.392658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.392627 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" containerID="cri-o://7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445" gracePeriod=30 Oct 30 20:17:26.440607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.440474 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:17:26.440770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.440714 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" containerID="cri-o://0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851" gracePeriod=30 Oct 30 20:17:26.444339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.444312 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:17:26.444527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.444505 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" containerID="cri-o://266476d022bc6e9d61eca0968b82c610f289d583157932cb4eb7c40e2d171265" gracePeriod=30 Oct 30 20:17:26.448282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.447964 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:17:26.448282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.448144 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" containerID="cri-o://b53f3225fe2c648d8865199106a1eb4ff59acc525f0bbeaa016598aaae09ac6f" gracePeriod=30 Oct 30 20:17:26.706406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:26.706372 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" event={"ID":"ac7bd72a-66dc-489c-b646-ea06c987b9f8","Type":"ContainerStarted","Data":"24a60a7a324ec7d1694933efe467ea7d718d6507ed54509198aed10adf51cf7b"} Oct 30 20:17:27.716986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:27.716937 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" event={"ID":"ac7bd72a-66dc-489c-b646-ea06c987b9f8","Type":"ContainerStarted","Data":"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08"} Oct 30 20:17:27.717398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:27.717140 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:27.840238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:27.840175 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" podStartSLOduration=2.900278954 podStartE2EDuration="3.840155642s" totalImagesPullingTime="939.876688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:27.837082232 +0000 UTC m=+2142.680753860" watchObservedRunningTime="2026-10-30 20:17:27.840155642 +0000 UTC m=+2142.683827271" Oct 30 20:17:28.848829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:28.848794 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" podUID="4a81e100-93ff-40e6-9d59-6073deb36750" containerName="payload-processing" containerID="cri-o://afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77" gracePeriod=25 Oct 30 20:17:29.103837 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.103751 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:17:29.146828 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.146796 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:17:29.196888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.196856 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\") pod \"4a81e100-93ff-40e6-9d59-6073deb36750\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " Oct 30 20:17:29.197056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.197016 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\" (UniqueName: \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\") pod \"4a81e100-93ff-40e6-9d59-6073deb36750\" (UID: \"4a81e100-93ff-40e6-9d59-6073deb36750\") " Oct 30 20:17:29.197285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.197261 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume" pod "4a81e100-93ff-40e6-9d59-6073deb36750" (UID: "4a81e100-93ff-40e6-9d59-6073deb36750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.197535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.197519 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81e100-93ff-40e6-9d59-6073deb36750-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.198922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.198895 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs" pod "4a81e100-93ff-40e6-9d59-6073deb36750" (UID: "4a81e100-93ff-40e6-9d59-6073deb36750"). InnerVolumeSpecName "kube-api-access-8b2fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.214859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.214834 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:17:29.299559 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.299533 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b2fs\" (UniqueName: \"kubernetes.io/projected/4a81e100-93ff-40e6-9d59-6073deb36750-kube-api-access-8b2fs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.579536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.579501 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:29.731835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.731798 4524 generic.go:356] "Generic (PLEG): container finished" podID="4a81e100-93ff-40e6-9d59-6073deb36750" containerID="afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77" exitCode=0 Oct 30 20:17:29.732013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.731851 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" Oct 30 20:17:29.732013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.731900 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" event={"ID":"4a81e100-93ff-40e6-9d59-6073deb36750","Type":"ContainerDied","Data":"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77"} Oct 30 20:17:29.732013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.731967 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx" event={"ID":"4a81e100-93ff-40e6-9d59-6073deb36750","Type":"ContainerDied","Data":"fc47eb5b14946b0de3fd85e5c81ca68f0c087da01d3a74888cb0fcbc9f6bfbc6"} Oct 30 20:17:29.732013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.731998 4524 scope.go:122] "RemoveContainer" containerID="afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77" Oct 30 20:17:29.853770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.851274 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:29.911729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.907027 4524 scope.go:122] "RemoveContainer" containerID="afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77" Oct 30 20:17:29.911729 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:29.911122 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77\": container with ID starting with afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77 not found: ID does not exist" containerID="afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77" Oct 30 20:17:29.911729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.911169 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77"} err="failed to get container status \"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77\": rpc error: code = NotFound desc = could not find container \"afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77\": container with ID starting with afbab48adcc7eee04a10e35943585bdc4a199ac632776f9cc380e45793bd9c77 not found: ID does not exist" Oct 30 20:17:29.956549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.953828 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:17:29.956549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:29.954799 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3736ef9-69cff7cb7d-vqmfx"] Oct 30 20:17:30.630465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:30.630424 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:31.302322 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.302284 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a81e100-93ff-40e6-9d59-6073deb36750" path="/var/lib/kubelet/pods/4a81e100-93ff-40e6-9d59-6073deb36750/volumes" Oct 30 20:17:31.746320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.746290 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:17:31.763197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.763171 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:17:31.776624 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776593 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.776817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776642 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.776817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776694 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.776817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776796 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776834 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776865 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.777012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776933 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776963 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.777012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.776989 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777060 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777105 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777160 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.777279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777249 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777341 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.777490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777403 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.777490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.777452 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.778088 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.778060 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\") pod \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\" (UID: \"d6c1367f-7dc1-45c3-a6dc-49a22793714c\") " Oct 30 20:17:31.778190 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.778109 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\") pod \"4ad1b637-ba01-4463-965a-7721df84b194\" (UID: \"4ad1b637-ba01-4463-965a-7721df84b194\") " Oct 30 20:17:31.779748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.779718 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.779935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.779910 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.780461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.780436 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.780735 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.781028 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.781301 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.781575 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.781924 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.783903 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.784865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.784771 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.785735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785605 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.785836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785788 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.785836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785808 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.785836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785824 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785838 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785852 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785868 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1b637-ba01-4463-965a-7721df84b194-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785881 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1b637-ba01-4463-965a-7721df84b194-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785895 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785908 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785922 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.785936 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.786350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.786208 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.786926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.786905 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.789070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.789041 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "kube-api-access-tvzkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.790263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.790238 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:31.790674 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.790643 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.790787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.790690 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m" pod "4ad1b637-ba01-4463-965a-7721df84b194" (UID: "4ad1b637-ba01-4463-965a-7721df84b194"). InnerVolumeSpecName "kube-api-access-66q7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.792265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.792232 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token" pod "d6c1367f-7dc1-45c3-a6dc-49a22793714c" (UID: "d6c1367f-7dc1-45c3-a6dc-49a22793714c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.793221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.793194 4524 generic.go:356] "Generic (PLEG): container finished" podID="4ad1b637-ba01-4463-965a-7721df84b194" containerID="7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445" exitCode=0 Oct 30 20:17:31.793319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.793258 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" Oct 30 20:17:31.793319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.793292 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" event={"ID":"4ad1b637-ba01-4463-965a-7721df84b194","Type":"ContainerDied","Data":"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445"} Oct 30 20:17:31.793510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.793375 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq" event={"ID":"4ad1b637-ba01-4463-965a-7721df84b194","Type":"ContainerDied","Data":"8ce3195eb3c5b203c2b71cec742874b27bfbb39af82afa33fa1064b02e16a8c4"} Oct 30 20:17:31.793510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.793421 4524 scope.go:122] "RemoveContainer" containerID="7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445" Oct 30 20:17:31.795855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.795833 4524 generic.go:356] "Generic (PLEG): container finished" podID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerID="266476d022bc6e9d61eca0968b82c610f289d583157932cb4eb7c40e2d171265" exitCode=0 Oct 30 20:17:31.796008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.795928 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" event={"ID":"3fb338c1-09b6-481a-b2a2-5d9302620681","Type":"ContainerDied","Data":"266476d022bc6e9d61eca0968b82c610f289d583157932cb4eb7c40e2d171265"} Oct 30 20:17:31.797280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.797261 4524 generic.go:356] "Generic (PLEG): container finished" podID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerID="0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851" exitCode=0 Oct 30 20:17:31.797399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.797325 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" event={"ID":"d6c1367f-7dc1-45c3-a6dc-49a22793714c","Type":"ContainerDied","Data":"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851"} Oct 30 20:17:31.797399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.797364 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" event={"ID":"d6c1367f-7dc1-45c3-a6dc-49a22793714c","Type":"ContainerDied","Data":"2011876388f0e77e41fca44b12cc2d5886be760892346a56a1152289519a3b3d"} Oct 30 20:17:31.797585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.797445 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw" Oct 30 20:17:31.803142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.802082 4524 generic.go:356] "Generic (PLEG): container finished" podID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerID="b53f3225fe2c648d8865199106a1eb4ff59acc525f0bbeaa016598aaae09ac6f" exitCode=0 Oct 30 20:17:31.803142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.802152 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" event={"ID":"0807c7d0-ad5b-4740-bfee-03ac643226e9","Type":"ContainerDied","Data":"b53f3225fe2c648d8865199106a1eb4ff59acc525f0bbeaa016598aaae09ac6f"} Oct 30 20:17:31.819952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.819931 4524 scope.go:122] "RemoveContainer" containerID="7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445" Oct 30 20:17:31.820249 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:31.820232 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445\": container with ID starting with 7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445 not found: ID does not exist" containerID="7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445" Oct 30 20:17:31.820306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.820258 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445"} err="failed to get container status \"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445\": rpc error: code = NotFound desc = could not find container \"7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445\": container with ID starting with 7376b1c8cc05fc49bf90059f90d2e9876cd727fd968ed06f71ca109da3abb445 not found: ID does not exist" Oct 30 20:17:31.820306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.820275 4524 scope.go:122] "RemoveContainer" containerID="0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851" Oct 30 20:17:31.824758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.824738 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:17:31.855117 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.855096 4524 scope.go:122] "RemoveContainer" containerID="0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851" Oct 30 20:17:31.855375 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:31.855355 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851\": container with ID starting with 0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851 not found: ID does not exist" containerID="0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851" Oct 30 20:17:31.855431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.855384 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851"} err="failed to get container status \"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851\": rpc error: code = NotFound desc = could not find container \"0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851\": container with ID starting with 0d290e05196aeb78ced46c0ad11751f754842b9ca71f03b3210f7190541be851 not found: ID does not exist" Oct 30 20:17:31.859516 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.859496 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:17:31.886870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.886844 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.886961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.886908 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.886957 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.886999 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887062 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887094 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887062 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887166 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887192 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887256 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887296 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887257 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887306 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887333 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887403 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887448 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887453 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887462 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887487 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887619 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887662 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\") pod \"3fb338c1-09b6-481a-b2a2-5d9302620681\" (UID: \"3fb338c1-09b6-481a-b2a2-5d9302620681\") " Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887691 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887731 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887744 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887745 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887783 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887839 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\") pod \"0807c7d0-ad5b-4740-bfee-03ac643226e9\" (UID: \"0807c7d0-ad5b-4740-bfee-03ac643226e9\") " Oct 30 20:17:31.887967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.887926 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.888460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888419 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66q7m\" (UniqueName: \"kubernetes.io/projected/4ad1b637-ba01-4463-965a-7721df84b194-kube-api-access-66q7m\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888438 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzkq\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-kube-api-access-tvzkq\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888453 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888468 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888481 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c1367f-7dc1-45c3-a6dc-49a22793714c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888494 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888508 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888521 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888535 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888549 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888564 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1b637-ba01-4463-965a-7721df84b194-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888578 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0807c7d0-ad5b-4740-bfee-03ac643226e9-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888591 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888605 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.888623 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888618 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb338c1-09b6-481a-b2a2-5d9302620681-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.890021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.889692 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:31.890021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.888632 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.890021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.889775 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.890399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.890370 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.890880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.890851 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.890973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.890928 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.890973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.890951 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "kube-api-access-57k7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.891120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.891104 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo" pod "0807c7d0-ad5b-4740-bfee-03ac643226e9" (UID: "0807c7d0-ad5b-4740-bfee-03ac643226e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:31.891419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.891392 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "kube-api-access-5zv52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:31.891524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.891494 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy" pod "3fb338c1-09b6-481a-b2a2-5d9302620681" (UID: "3fb338c1-09b6-481a-b2a2-5d9302620681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:31.991615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991580 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991609 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zv52\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-kube-api-access-5zv52\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991620 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991630 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991641 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57k7q\" (UniqueName: \"kubernetes.io/projected/0807c7d0-ad5b-4740-bfee-03ac643226e9-kube-api-access-57k7q\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991650 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0807c7d0-ad5b-4740-bfee-03ac643226e9-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991659 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:31.991973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:31.991668 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb338c1-09b6-481a-b2a2-5d9302620681-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:32.360499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.360461 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:17:32.370183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.370155 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dllfw"] Oct 30 20:17:32.487575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.487544 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:17:32.491998 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.491971 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xpbtq"] Oct 30 20:17:32.809822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.809785 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" event={"ID":"3fb338c1-09b6-481a-b2a2-5d9302620681","Type":"ContainerDied","Data":"6b55f8e220681d5e88300a78285a039966dae87c2c879744f9fd5806e0c3165e"} Oct 30 20:17:32.809822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.809821 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8" Oct 30 20:17:32.810128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.809852 4524 scope.go:122] "RemoveContainer" containerID="266476d022bc6e9d61eca0968b82c610f289d583157932cb4eb7c40e2d171265" Oct 30 20:17:32.812520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.812497 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" Oct 30 20:17:32.812637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.812524 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh" event={"ID":"0807c7d0-ad5b-4740-bfee-03ac643226e9","Type":"ContainerDied","Data":"86d6c38a52999f922551407b3412fc67378af6a7906adcbab00b5c32895ddde6"} Oct 30 20:17:32.820860 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.820636 4524 scope.go:122] "RemoveContainer" containerID="b53f3225fe2c648d8865199106a1eb4ff59acc525f0bbeaa016598aaae09ac6f" Oct 30 20:17:32.980332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.980302 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:17:32.985503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:32.985475 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5vkt8"] Oct 30 20:17:33.097750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.097673 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:17:33.103608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.103584 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9rckh"] Oct 30 20:17:33.306835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.306799 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" path="/var/lib/kubelet/pods/0807c7d0-ad5b-4740-bfee-03ac643226e9/volumes" Oct 30 20:17:33.307267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.307252 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" path="/var/lib/kubelet/pods/3fb338c1-09b6-481a-b2a2-5d9302620681/volumes" Oct 30 20:17:33.307679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.307667 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad1b637-ba01-4463-965a-7721df84b194" path="/var/lib/kubelet/pods/4ad1b637-ba01-4463-965a-7721df84b194/volumes" Oct 30 20:17:33.308436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.308421 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" path="/var/lib/kubelet/pods/d6c1367f-7dc1-45c3-a6dc-49a22793714c/volumes" Oct 30 20:17:33.729029 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:33.729001 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:17:34.104148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.104042 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" podUID="2b8a200a-3f8b-4daf-9728-506fc78ed149" containerName="payload-processing" containerID="cri-o://2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d" gracePeriod=25 Oct 30 20:17:34.215925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.215883 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" podUID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" containerName="payload-pre-processing" containerID="cri-o://42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb" gracePeriod=25 Oct 30 20:17:34.394575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.394547 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:17:34.500423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.500403 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:17:34.523401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.523379 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\") pod \"2b8a200a-3f8b-4daf-9728-506fc78ed149\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " Oct 30 20:17:34.523546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.523532 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\" (UniqueName: \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\") pod \"2b8a200a-3f8b-4daf-9728-506fc78ed149\" (UID: \"2b8a200a-3f8b-4daf-9728-506fc78ed149\") " Oct 30 20:17:34.523693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.523673 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume" pod "2b8a200a-3f8b-4daf-9728-506fc78ed149" (UID: "2b8a200a-3f8b-4daf-9728-506fc78ed149"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:34.523895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.523874 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a200a-3f8b-4daf-9728-506fc78ed149-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:34.525513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.525493 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9" pod "2b8a200a-3f8b-4daf-9728-506fc78ed149" (UID: "2b8a200a-3f8b-4daf-9728-506fc78ed149"). InnerVolumeSpecName "kube-api-access-qhjm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:34.625144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.625075 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\" (UniqueName: \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\") pod \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " Oct 30 20:17:34.625288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.625260 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\") pod \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\" (UID: \"60306054-eae8-4fb9-b1b1-7841ff67ebb6\") " Oct 30 20:17:34.625576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.625559 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjm9\" (UniqueName: \"kubernetes.io/projected/2b8a200a-3f8b-4daf-9728-506fc78ed149-kube-api-access-qhjm9\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:34.625672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.625592 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume" pod "60306054-eae8-4fb9-b1b1-7841ff67ebb6" (UID: "60306054-eae8-4fb9-b1b1-7841ff67ebb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:34.627310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.627282 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr" pod "60306054-eae8-4fb9-b1b1-7841ff67ebb6" (UID: "60306054-eae8-4fb9-b1b1-7841ff67ebb6"). InnerVolumeSpecName "kube-api-access-f98sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:34.726848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.726817 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60306054-eae8-4fb9-b1b1-7841ff67ebb6-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:34.726848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.726852 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f98sr\" (UniqueName: \"kubernetes.io/projected/60306054-eae8-4fb9-b1b1-7841ff67ebb6-kube-api-access-f98sr\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:34.830601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.830567 4524 generic.go:356] "Generic (PLEG): container finished" podID="2b8a200a-3f8b-4daf-9728-506fc78ed149" containerID="2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d" exitCode=0 Oct 30 20:17:34.831065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.830622 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" Oct 30 20:17:34.831065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.830652 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" event={"ID":"2b8a200a-3f8b-4daf-9728-506fc78ed149","Type":"ContainerDied","Data":"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d"} Oct 30 20:17:34.831065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.830729 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt" event={"ID":"2b8a200a-3f8b-4daf-9728-506fc78ed149","Type":"ContainerDied","Data":"97425f88e2410b3cee8de0eeda9da2c6d3915f86c6b44e98ae6b4be1807615d7"} Oct 30 20:17:34.831065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.830759 4524 scope.go:122] "RemoveContainer" containerID="2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d" Oct 30 20:17:34.832375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.832351 4524 generic.go:356] "Generic (PLEG): container finished" podID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" containerID="42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb" exitCode=0 Oct 30 20:17:34.832531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.832403 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" Oct 30 20:17:34.832531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.832417 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" event={"ID":"60306054-eae8-4fb9-b1b1-7841ff67ebb6","Type":"ContainerDied","Data":"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb"} Oct 30 20:17:34.832531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.832456 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr" event={"ID":"60306054-eae8-4fb9-b1b1-7841ff67ebb6","Type":"ContainerDied","Data":"ea12c6bbaf28715d34bf74319c71f412339af39ee0f6610f3013c7d5cd58d93c"} Oct 30 20:17:34.842421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.842405 4524 scope.go:122] "RemoveContainer" containerID="2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d" Oct 30 20:17:34.842729 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:34.842671 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d\": container with ID starting with 2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d not found: ID does not exist" containerID="2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d" Oct 30 20:17:34.842862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.842722 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d"} err="failed to get container status \"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d\": rpc error: code = NotFound desc = could not find container \"2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d\": container with ID starting with 2ebd704c657a12aa6c7d2ff211e289e1e7206dd4730fdae2d291e994a2a6813d not found: ID does not exist" Oct 30 20:17:34.842862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.842745 4524 scope.go:122] "RemoveContainer" containerID="42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb" Oct 30 20:17:34.851646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.851625 4524 scope.go:122] "RemoveContainer" containerID="42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb" Oct 30 20:17:34.851954 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:34.851937 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb\": container with ID starting with 42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb not found: ID does not exist" containerID="42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb" Oct 30 20:17:34.852007 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.851961 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb"} err="failed to get container status \"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb\": rpc error: code = NotFound desc = could not find container \"42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb\": container with ID starting with 42520b1e883eb0916647248c55f000959aa8f076aabce30b408d24ad0f1024fb not found: ID does not exist" Oct 30 20:17:34.997270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:34.997240 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:17:35.001196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.001171 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fcfa-74cdf5c475-ssspr"] Oct 30 20:17:35.110321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.110289 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:17:35.115588 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.115563 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fcfa-65bf7cb594-k4jtt"] Oct 30 20:17:35.307625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.307552 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8a200a-3f8b-4daf-9728-506fc78ed149" path="/var/lib/kubelet/pods/2b8a200a-3f8b-4daf-9728-506fc78ed149/volumes" Oct 30 20:17:35.308180 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.308160 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" path="/var/lib/kubelet/pods/60306054-eae8-4fb9-b1b1-7841ff67ebb6/volumes" Oct 30 20:17:35.645776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.639793 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.855174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:35.855138 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.569377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.569333 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.585755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.575006 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9161-62c8aa"] Oct 30 20:17:40.587240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587201 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" Oct 30 20:17:40.587435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587420 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" Oct 30 20:17:40.587559 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587548 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" Oct 30 20:17:40.587652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587642 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" Oct 30 20:17:40.587790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587779 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4a81e100-93ff-40e6-9d59-6073deb36750" containerName="payload-processing" Oct 30 20:17:40.587895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.587884 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a81e100-93ff-40e6-9d59-6073deb36750" containerName="payload-processing" Oct 30 20:17:40.588038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588027 4524 cpu_manager.go:414] "containerMap: removing container" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" Oct 30 20:17:40.588135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588124 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" Oct 30 20:17:40.588237 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588228 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" Oct 30 20:17:40.588328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588318 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" Oct 30 20:17:40.588413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588404 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2b8a200a-3f8b-4daf-9728-506fc78ed149" containerName="payload-processing" Oct 30 20:17:40.588506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588495 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8a200a-3f8b-4daf-9728-506fc78ed149" containerName="payload-processing" Oct 30 20:17:40.588611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588602 4524 cpu_manager.go:414] "containerMap: removing container" podUID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" containerName="payload-pre-processing" Oct 30 20:17:40.588719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.588692 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" containerName="payload-pre-processing" Oct 30 20:17:40.589022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589001 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad1b637-ba01-4463-965a-7721df84b194" containerName="istio-proxy" Oct 30 20:17:40.589139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589129 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb338c1-09b6-481a-b2a2-5d9302620681" containerName="istio-proxy" Oct 30 20:17:40.589245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589235 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a81e100-93ff-40e6-9d59-6073deb36750" containerName="payload-processing" Oct 30 20:17:40.589340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589331 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="0807c7d0-ad5b-4740-bfee-03ac643226e9" containerName="istio-proxy" Oct 30 20:17:40.589438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589429 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c1367f-7dc1-45c3-a6dc-49a22793714c" containerName="istio-proxy" Oct 30 20:17:40.589523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589515 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8a200a-3f8b-4daf-9728-506fc78ed149" containerName="payload-processing" Oct 30 20:17:40.589605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.589596 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="60306054-eae8-4fb9-b1b1-7841ff67ebb6" containerName="payload-pre-processing" Oct 30 20:17:40.610379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.610352 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9161-62c8aa"] Oct 30 20:17:40.610730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.610640 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:40.712433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.712404 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nts8c\"" Oct 30 20:17:40.720125 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.720100 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcf4l\" (UniqueName: \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\") pod \"test-curl-9161-62c8aa\" (UID: \"6072c8d9-3637-49aa-862b-cada47d3c2c1\") " pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:40.821684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.821618 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcf4l\" (UniqueName: \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\") pod \"test-curl-9161-62c8aa\" (UID: \"6072c8d9-3637-49aa-862b-cada47d3c2c1\") " pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:40.883052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.883023 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcf4l\" (UniqueName: \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\") pod \"test-curl-9161-62c8aa\" (UID: \"6072c8d9-3637-49aa-862b-cada47d3c2c1\") " pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:40.932443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:40.932418 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:41.308727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:41.308685 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9161-62c8aa"] Oct 30 20:17:41.309291 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:17:41.309260 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6072c8d9_3637_49aa_862b_cada47d3c2c1.slice/crio-d9ab468e0d586a28400082e79ba1a99ac6e729d4197b228e8c8784b988dd19eb WatchSource:0}: Error finding container d9ab468e0d586a28400082e79ba1a99ac6e729d4197b228e8c8784b988dd19eb: Status 404 returned error can't find the container with id d9ab468e0d586a28400082e79ba1a99ac6e729d4197b228e8c8784b988dd19eb Oct 30 20:17:41.882938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:41.882902 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9161-62c8aa" event={"ID":"6072c8d9-3637-49aa-862b-cada47d3c2c1","Type":"ContainerStarted","Data":"e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee"} Oct 30 20:17:41.882938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:41.882943 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9161-62c8aa" event={"ID":"6072c8d9-3637-49aa-862b-cada47d3c2c1","Type":"ContainerStarted","Data":"d9ab468e0d586a28400082e79ba1a99ac6e729d4197b228e8c8784b988dd19eb"} Oct 30 20:17:42.009668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:42.009616 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9161-62c8aa" podStartSLOduration=2.009604177 podStartE2EDuration="2.009604177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:42.007221128 +0000 UTC m=+2156.850892758" watchObservedRunningTime="2026-10-30 20:17:42.009604177 +0000 UTC m=+2156.853275805" Oct 30 20:17:42.787565 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:42.787524 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:17:43.569241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:43.569206 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9161-62c8aa"] Oct 30 20:17:43.574875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:43.574840 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9161-62c8aa"] Oct 30 20:17:43.898071 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:43.898030 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9161-62c8aa" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" containerName="test-curl-9161-62c8aa" containerID="cri-o://e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee" gracePeriod=2 Oct 30 20:17:43.949534 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:43.949502 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9161-62c8aa\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:45.347271 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:45.347234 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9161-62c8aa\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:45.550074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:45.550034 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.165526 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.165457 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:46.209739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.209706 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\" (UniqueName: \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\") pod \"6072c8d9-3637-49aa-862b-cada47d3c2c1\" (UID: \"6072c8d9-3637-49aa-862b-cada47d3c2c1\") " Oct 30 20:17:46.211807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.211781 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l" pod "6072c8d9-3637-49aa-862b-cada47d3c2c1" (UID: "6072c8d9-3637-49aa-862b-cada47d3c2c1"). InnerVolumeSpecName "kube-api-access-jcf4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:46.311263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.311223 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcf4l\" (UniqueName: \"kubernetes.io/projected/6072c8d9-3637-49aa-862b-cada47d3c2c1-kube-api-access-jcf4l\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:46.614644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.614520 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.918474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.918436 4524 generic.go:356] "Generic (PLEG): container finished" podID="6072c8d9-3637-49aa-862b-cada47d3c2c1" containerID="e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee" exitCode=137 Oct 30 20:17:46.918620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.918479 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:46.918620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.918537 4524 scope.go:122] "RemoveContainer" containerID="e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee" Oct 30 20:17:46.928379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.928363 4524 scope.go:122] "RemoveContainer" containerID="e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee" Oct 30 20:17:46.928622 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:46.928601 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee\": container with ID starting with e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee not found: ID does not exist" containerID="e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee" Oct 30 20:17:46.928686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:46.928632 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee"} err="failed to get container status \"e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee\": rpc error: code = NotFound desc = could not find container \"e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee\": container with ID starting with e17aa1291dd356a574331e8ef3f2ede0c8911656afc32cfd9b213f37737e18ee not found: ID does not exist" Oct 30 20:17:46.978723 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:17:46.978682 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9161-62c8aa\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" pod="opendatahub/test-curl-9161-62c8aa" Oct 30 20:17:47.301032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:47.300955 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" path="/var/lib/kubelet/pods/6072c8d9-3637-49aa-862b-cada47d3c2c1/volumes" Oct 30 20:17:48.169564 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.169532 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:17:48.170244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.170226 4524 cpu_manager.go:414] "containerMap: removing container" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" containerName="test-curl-9161-62c8aa" Oct 30 20:17:48.170288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.170246 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" containerName="test-curl-9161-62c8aa" Oct 30 20:17:48.170389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.170379 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="6072c8d9-3637-49aa-862b-cada47d3c2c1" containerName="test-curl-9161-62c8aa" Oct 30 20:17:48.175435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.175415 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234161 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234210 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234374 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234419 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234604 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489cg\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234674 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234662 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234814 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.234876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234857 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.235009 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.234985 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.278670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.278630 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37732de7-openshift-default-dockercfg-g5kkn\"" Oct 30 20:17:48.285928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.285896 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:17:48.335897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.335865 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.335928 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.335977 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336026 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336046 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336135 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489cg\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336257 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336182 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336284 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336293 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336325 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336385 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336466 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.336601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.336578 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.337115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.337088 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.338539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.338516 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.338647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.338620 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.397304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.397281 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.397435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.397379 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489cg\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.490807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.490729 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:48.925865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.925837 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:17:48.926844 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:17:48.926818 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e0f1dd_5699_4b55_bf54_d248a80c132c.slice/crio-20e49d40d081ce7206e82830eef6a3c8ddb378cefb62ec5fead82e615a5477f0 WatchSource:0}: Error finding container 20e49d40d081ce7206e82830eef6a3c8ddb378cefb62ec5fead82e615a5477f0: Status 404 returned error can't find the container with id 20e49d40d081ce7206e82830eef6a3c8ddb378cefb62ec5fead82e615a5477f0 Oct 30 20:17:48.929965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.929924 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:48.930040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.930012 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:48.930085 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:48.930053 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:49.942457 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:49.942421 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" event={"ID":"85e0f1dd-5699-4b55-bf54-d248a80c132c","Type":"ContainerStarted","Data":"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437"} Oct 30 20:17:49.942862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:49.942480 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" event={"ID":"85e0f1dd-5699-4b55-bf54-d248a80c132c","Type":"ContainerStarted","Data":"20e49d40d081ce7206e82830eef6a3c8ddb378cefb62ec5fead82e615a5477f0"} Oct 30 20:17:50.115592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:50.115534 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podStartSLOduration=2.115519389 podStartE2EDuration="2.115519389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:50.112355399 +0000 UTC m=+2164.956027029" watchObservedRunningTime="2026-10-30 20:17:50.115519389 +0000 UTC m=+2164.959191017" Oct 30 20:17:50.491216 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:50.491185 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:50.493367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:50.493343 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:50.493479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:50.493395 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:50.728305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:50.728266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:51.491406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:51.491370 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:51.491953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:51.491433 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:52.491305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:52.491273 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:52.491522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:52.491325 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:53.491865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:53.491825 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:53.492361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:53.492321 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:54.491922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:54.491871 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:54.492363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:54.491934 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:55.491199 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:55.491163 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:17:55.491382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:55.491216 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 20:17:55.802953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:55.802845 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:56.132926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.132850 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:17:56.453244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.453205 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:56.505933 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.505894 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:56.509557 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.509525 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:56.510357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.510333 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:17:56.691235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:17:56.691199 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.037163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.037129 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:18:01.042621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.042590 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.101215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.101181 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:18:01.133714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.133672 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" containerID="cri-o://ea701f6e43d9cef5ca1059f79178ebef2770377c6cf18b4d1e3c9c83d6771a90" gracePeriod=25 Oct 30 20:18:01.136426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.136371 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.136683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.136658 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.136970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.136943 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2l4\" (UniqueName: \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.137083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.137009 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.177106 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.177076 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-25cc2f-serving-cert\"" Oct 30 20:18:01.238725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.238672 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.238888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.238806 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.239038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.239018 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.239086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.239058 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2l4\" (UniqueName: \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.239184 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:01.239164 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-serving-cert: secret "maas-api-e2e-api-a-25cc2f-serving-cert" not found Oct 30 20:18:01.239270 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:01.239257 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls podName:59b7c55e-6ee8-433c-aab5-1c808334c9b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:18:01.739235873 +0000 UTC m=+2176.582907494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls") pod "maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" (UID: "59b7c55e-6ee8-433c-aab5-1c808334c9b8") : secret "maas-api-e2e-api-a-25cc2f-serving-cert" not found Oct 30 20:18:01.239555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.239534 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.241401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.241380 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.310193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.310110 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2l4\" (UniqueName: \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.415857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.415819 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.671722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.671279 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.756724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.748456 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.792733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.785179 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") pod \"maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:01.890990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.890960 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.978935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:01.978845 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:02.399680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:02.399652 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:18:02.401317 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:18:02.401290 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b7c55e_6ee8_433c_aab5_1c808334c9b8.slice/crio-5e315531d2a46392b24c492026c832408e40f0d085c95fb31563c4a7f9841995 WatchSource:0}: Error finding container 5e315531d2a46392b24c492026c832408e40f0d085c95fb31563c4a7f9841995: Status 404 returned error can't find the container with id 5e315531d2a46392b24c492026c832408e40f0d085c95fb31563c4a7f9841995 Oct 30 20:18:02.478967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:02.478874 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:03.143304 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:03.143269 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" event={"ID":"59b7c55e-6ee8-433c-aab5-1c808334c9b8","Type":"ContainerStarted","Data":"5e315531d2a46392b24c492026c832408e40f0d085c95fb31563c4a7f9841995"} Oct 30 20:18:04.152973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:04.152928 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" event={"ID":"59b7c55e-6ee8-433c-aab5-1c808334c9b8","Type":"ContainerStarted","Data":"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a"} Oct 30 20:18:04.153458 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:04.153040 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:04.271284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:04.271216 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" podStartSLOduration=3.43421951 podStartE2EDuration="4.271199228s" totalImagesPullingTime="836.979718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:04.271048283 +0000 UTC m=+2179.114719914" watchObservedRunningTime="2026-10-30 20:18:04.271199228 +0000 UTC m=+2179.114870857" Oct 30 20:18:06.100185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.100131 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:18:06.102019 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.101969 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" podUID="7e314883-d870-4fa3-9396-909f32ddae37" containerName="maas-api" containerID="cri-o://e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad" gracePeriod=30 Oct 30 20:18:06.220493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.220461 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:06.568144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.563201 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:18:06.691499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.689442 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\") pod \"7e314883-d870-4fa3-9396-909f32ddae37\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " Oct 30 20:18:06.691499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.689503 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") pod \"7e314883-d870-4fa3-9396-909f32ddae37\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " Oct 30 20:18:06.691499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.689652 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\") pod \"7e314883-d870-4fa3-9396-909f32ddae37\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " Oct 30 20:18:06.691499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.689750 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\" (UniqueName: \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\") pod \"7e314883-d870-4fa3-9396-909f32ddae37\" (UID: \"7e314883-d870-4fa3-9396-909f32ddae37\") " Oct 30 20:18:06.692683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.692647 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca" pod "7e314883-d870-4fa3-9396-909f32ddae37" (UID: "7e314883-d870-4fa3-9396-909f32ddae37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:06.695346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.695313 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs" pod "7e314883-d870-4fa3-9396-909f32ddae37" (UID: "7e314883-d870-4fa3-9396-909f32ddae37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:06.706911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.706879 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls" pod "7e314883-d870-4fa3-9396-909f32ddae37" (UID: "7e314883-d870-4fa3-9396-909f32ddae37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:06.707313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.707287 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb" pod "7e314883-d870-4fa3-9396-909f32ddae37" (UID: "7e314883-d870-4fa3-9396-909f32ddae37"). InnerVolumeSpecName "kube-api-access-4btxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:06.793038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.792326 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:06.793038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.792367 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e314883-d870-4fa3-9396-909f32ddae37-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:06.793038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.792405 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e314883-d870-4fa3-9396-909f32ddae37-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:06.793038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:06.792419 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btxb\" (UniqueName: \"kubernetes.io/projected/7e314883-d870-4fa3-9396-909f32ddae37-kube-api-access-4btxb\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:07.192013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.191865 4524 generic.go:356] "Generic (PLEG): container finished" podID="7e314883-d870-4fa3-9396-909f32ddae37" containerID="e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad" exitCode=0 Oct 30 20:18:07.192013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.191932 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" Oct 30 20:18:07.193285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.192076 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" event={"ID":"7e314883-d870-4fa3-9396-909f32ddae37","Type":"ContainerDied","Data":"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad"} Oct 30 20:18:07.193285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.192119 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn" event={"ID":"7e314883-d870-4fa3-9396-909f32ddae37","Type":"ContainerDied","Data":"8cd1f6407e3762ed9dd3d364555f67b0f95b68ad1ecf160a92ac36382b5bb5c6"} Oct 30 20:18:07.193285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.192145 4524 scope.go:122] "RemoveContainer" containerID="e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad" Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.194714 4524 generic.go:356] "Generic (PLEG): container finished" podID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerID="4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9" exitCode=0 Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.194780 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerDied","Data":"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9"} Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.195200 4524 scope.go:122] "RemoveContainer" containerID="4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9" Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.196917 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.196992 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:07.198246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.197025 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:07.203859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.203788 4524 scope.go:122] "RemoveContainer" containerID="e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad" Oct 30 20:18:07.204647 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:07.204618 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad\": container with ID starting with e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad not found: ID does not exist" containerID="e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad" Oct 30 20:18:07.204840 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.204656 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad"} err="failed to get container status \"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad\": rpc error: code = NotFound desc = could not find container \"e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad\": container with ID starting with e9beac80d2eeb6be43b2acf8733282aaf8118058debead8659332c943ceff1ad not found: ID does not exist" Oct 30 20:18:07.495588 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.495554 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:18:07.507911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.507874 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7961ed-7b7769f4b-smkbn"] Oct 30 20:18:07.973412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:07.973379 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:08.203291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.203256 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerStarted","Data":"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d"} Oct 30 20:18:08.216678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.216645 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:08.427641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.427597 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:08.997496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.997468 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:18:08.998740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.998681 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:18:08.998873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:08.998755 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:18:09.070760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:09.070513 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:09.303416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:09.303334 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e314883-d870-4fa3-9396-909f32ddae37" path="/var/lib/kubelet/pods/7e314883-d870-4fa3-9396-909f32ddae37/volumes" Oct 30 20:18:09.998075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:09.998041 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:18:09.998265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:09.998092 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:18:10.173924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:10.173888 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:18:10.998018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:10.997979 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:18:10.998447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:10.998054 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:18:11.997645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:11.997616 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:18:11.997851 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:11.997679 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:18:12.206673 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.206641 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:18:12.325942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.325867 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:18:12.477153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.477096 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:12.636434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.636345 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:12.715853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.715819 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:18:12.847123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.847080 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:12.909587 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:12.909554 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:18:12.998259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.998216 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:18:12.998474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:12.998270 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:18:13.888903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:13.888861 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:14.008012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:14.007982 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:18:14.011865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:14.011834 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:18:15.255435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:15.255381 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:16.437069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:16.437022 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:16.674620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:16.674578 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:17.002722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.000748 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:17.207646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.207602 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" podUID="8640f882-eb60-4ee3-a0f1-2940dff09143" containerName="payload-processing" containerID="cri-o://547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116" gracePeriod=25 Oct 30 20:18:17.327012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.326984 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" podUID="da281092-bd97-41ef-a470-a47e285a7da4" containerName="payload-pre-processing" containerID="cri-o://d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a" gracePeriod=25 Oct 30 20:18:17.511193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.511173 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:18:17.605128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.605041 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\" (UniqueName: \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\") pod \"8640f882-eb60-4ee3-a0f1-2940dff09143\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " Oct 30 20:18:17.605269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.605150 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\") pod \"8640f882-eb60-4ee3-a0f1-2940dff09143\" (UID: \"8640f882-eb60-4ee3-a0f1-2940dff09143\") " Oct 30 20:18:17.605894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.605860 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume" pod "8640f882-eb60-4ee3-a0f1-2940dff09143" (UID: "8640f882-eb60-4ee3-a0f1-2940dff09143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:17.607501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.607473 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt" pod "8640f882-eb60-4ee3-a0f1-2940dff09143" (UID: "8640f882-eb60-4ee3-a0f1-2940dff09143"). InnerVolumeSpecName "kube-api-access-cq6jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:17.626263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.626240 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:18:17.706988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.706953 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\" (UniqueName: \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\") pod \"da281092-bd97-41ef-a470-a47e285a7da4\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " Oct 30 20:18:17.707156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.707050 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\") pod \"da281092-bd97-41ef-a470-a47e285a7da4\" (UID: \"da281092-bd97-41ef-a470-a47e285a7da4\") " Oct 30 20:18:17.707425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.707399 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume" pod "da281092-bd97-41ef-a470-a47e285a7da4" (UID: "da281092-bd97-41ef-a470-a47e285a7da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:17.707578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.707563 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6jt\" (UniqueName: \"kubernetes.io/projected/8640f882-eb60-4ee3-a0f1-2940dff09143-kube-api-access-cq6jt\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.707643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.707579 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8640f882-eb60-4ee3-a0f1-2940dff09143-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.707643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.707588 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da281092-bd97-41ef-a470-a47e285a7da4-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.709215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.709192 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m" pod "da281092-bd97-41ef-a470-a47e285a7da4" (UID: "da281092-bd97-41ef-a470-a47e285a7da4"). InnerVolumeSpecName "kube-api-access-6vb2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:17.809223 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:17.809188 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vb2m\" (UniqueName: \"kubernetes.io/projected/da281092-bd97-41ef-a470-a47e285a7da4-kube-api-access-6vb2m\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:18.274022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.273988 4524 generic.go:356] "Generic (PLEG): container finished" podID="8640f882-eb60-4ee3-a0f1-2940dff09143" containerID="547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116" exitCode=0 Oct 30 20:18:18.274184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.274044 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" Oct 30 20:18:18.274184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.274041 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" event={"ID":"8640f882-eb60-4ee3-a0f1-2940dff09143","Type":"ContainerDied","Data":"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116"} Oct 30 20:18:18.274312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.274221 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx" event={"ID":"8640f882-eb60-4ee3-a0f1-2940dff09143","Type":"ContainerDied","Data":"fb39e9cdd49197f9a5d4d78f447a3191b4eae719f899f672e593ca88d919d703"} Oct 30 20:18:18.274312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.274259 4524 scope.go:122] "RemoveContainer" containerID="547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116" Oct 30 20:18:18.275653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.275606 4524 generic.go:356] "Generic (PLEG): container finished" podID="da281092-bd97-41ef-a470-a47e285a7da4" containerID="d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a" exitCode=0 Oct 30 20:18:18.276732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.275787 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" Oct 30 20:18:18.276732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.275877 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" event={"ID":"da281092-bd97-41ef-a470-a47e285a7da4","Type":"ContainerDied","Data":"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a"} Oct 30 20:18:18.276732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.275922 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx" event={"ID":"da281092-bd97-41ef-a470-a47e285a7da4","Type":"ContainerDied","Data":"5b160d77b1f2cf54c376b0e2646e714ebbe80fcea74ed17cf470fa2afa0eddb2"} Oct 30 20:18:18.286750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.286731 4524 scope.go:122] "RemoveContainer" containerID="547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116" Oct 30 20:18:18.287077 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:18.287051 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116\": container with ID starting with 547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116 not found: ID does not exist" containerID="547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116" Oct 30 20:18:18.287151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.287087 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116"} err="failed to get container status \"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116\": rpc error: code = NotFound desc = could not find container \"547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116\": container with ID starting with 547449747eaefc557f64c5e62baea04a5fbf4f2b5e1a6f61927d771f6d400116 not found: ID does not exist" Oct 30 20:18:18.287151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.287109 4524 scope.go:122] "RemoveContainer" containerID="d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a" Oct 30 20:18:18.299531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.299512 4524 scope.go:122] "RemoveContainer" containerID="d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a" Oct 30 20:18:18.299807 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:18.299783 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a\": container with ID starting with d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a not found: ID does not exist" containerID="d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a" Oct 30 20:18:18.299875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.299809 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a"} err="failed to get container status \"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a\": rpc error: code = NotFound desc = could not find container \"d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a\": container with ID starting with d44ca3337bdc1db4f714e100f1b4e9d217465146e05c7dda2f66457884dd367a not found: ID does not exist" Oct 30 20:18:18.442392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.442363 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:18:18.492046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.491973 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d2c70-6b9f5bb68d-m2ghx"] Oct 30 20:18:18.603694 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.603664 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:18:18.610855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:18.610830 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d2c70-b46c8845f-lvdgx"] Oct 30 20:18:19.300766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:19.300733 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8640f882-eb60-4ee3-a0f1-2940dff09143" path="/var/lib/kubelet/pods/8640f882-eb60-4ee3-a0f1-2940dff09143/volumes" Oct 30 20:18:19.301175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:19.301161 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da281092-bd97-41ef-a470-a47e285a7da4" path="/var/lib/kubelet/pods/da281092-bd97-41ef-a470-a47e285a7da4/volumes" Oct 30 20:18:19.567770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:19.566598 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.313062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.312835 4524 generic.go:356] "Generic (PLEG): container finished" podID="8f8869a5-210e-4527-8c8c-85e72233665e" containerID="ea701f6e43d9cef5ca1059f79178ebef2770377c6cf18b4d1e3c9c83d6771a90" exitCode=0 Oct 30 20:18:21.313062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.312963 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" event={"ID":"8f8869a5-210e-4527-8c8c-85e72233665e","Type":"ContainerDied","Data":"ea701f6e43d9cef5ca1059f79178ebef2770377c6cf18b4d1e3c9c83d6771a90"} Oct 30 20:18:21.420246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.420215 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:18:21.560735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.560684 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\") pod \"8f8869a5-210e-4527-8c8c-85e72233665e\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " Oct 30 20:18:21.560925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.560872 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\" (UniqueName: \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\") pod \"8f8869a5-210e-4527-8c8c-85e72233665e\" (UID: \"8f8869a5-210e-4527-8c8c-85e72233665e\") " Oct 30 20:18:21.561156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.561130 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume" pod "8f8869a5-210e-4527-8c8c-85e72233665e" (UID: "8f8869a5-210e-4527-8c8c-85e72233665e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:21.561973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.561944 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8869a5-210e-4527-8c8c-85e72233665e-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:21.563612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.563576 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98" pod "8f8869a5-210e-4527-8c8c-85e72233665e" (UID: "8f8869a5-210e-4527-8c8c-85e72233665e"). InnerVolumeSpecName "kube-api-access-6zp98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:21.663682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:21.663603 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zp98\" (UniqueName: \"kubernetes.io/projected/8f8869a5-210e-4527-8c8c-85e72233665e-kube-api-access-6zp98\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:22.321034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:22.321002 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" Oct 30 20:18:22.321462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:22.321004 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl" event={"ID":"8f8869a5-210e-4527-8c8c-85e72233665e","Type":"ContainerDied","Data":"9929b42b86231cdcfe774fb4811810f2b52360ec3020c18092054a0eb4999328"} Oct 30 20:18:22.321462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:22.321136 4524 scope.go:122] "RemoveContainer" containerID="ea701f6e43d9cef5ca1059f79178ebef2770377c6cf18b4d1e3c9c83d6771a90" Oct 30 20:18:22.499528 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:22.499495 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:18:22.504123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:22.504092 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-m66zl"] Oct 30 20:18:23.303589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:23.303551 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" path="/var/lib/kubelet/pods/8f8869a5-210e-4527-8c8c-85e72233665e/volumes" Oct 30 20:18:24.603542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:24.602325 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:28.710465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:28.710372 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:18:28.710965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:28.710617 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" podUID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" containerName="maas-api" containerID="cri-o://4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5" gracePeriod=30 Oct 30 20:18:28.962838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:28.961635 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:29.224768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.222154 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.226248 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.237296 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\") pod \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.237404 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") pod \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.237514 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\" (UniqueName: \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\") pod \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.237677 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\") pod \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\" (UID: \"48b890a8-5e32-4ddf-b802-0647ff7fe12a\") " Oct 30 20:18:29.247390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.243806 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca" pod "48b890a8-5e32-4ddf-b802-0647ff7fe12a" (UID: "48b890a8-5e32-4ddf-b802-0647ff7fe12a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:29.303958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.279384 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r" pod "48b890a8-5e32-4ddf-b802-0647ff7fe12a" (UID: "48b890a8-5e32-4ddf-b802-0647ff7fe12a"). InnerVolumeSpecName "kube-api-access-55s7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:29.303958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.302312 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls" pod "48b890a8-5e32-4ddf-b802-0647ff7fe12a" (UID: "48b890a8-5e32-4ddf-b802-0647ff7fe12a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:29.307991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.306171 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs" pod "48b890a8-5e32-4ddf-b802-0647ff7fe12a" (UID: "48b890a8-5e32-4ddf-b802-0647ff7fe12a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:29.323284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.321675 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:18:29.339475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.339292 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55s7r\" (UniqueName: \"kubernetes.io/projected/48b890a8-5e32-4ddf-b802-0647ff7fe12a-kube-api-access-55s7r\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:29.339475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.339319 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b890a8-5e32-4ddf-b802-0647ff7fe12a-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:29.339475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.339329 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:29.339475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.339339 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48b890a8-5e32-4ddf-b802-0647ff7fe12a-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:29.410448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.410415 4524 generic.go:356] "Generic (PLEG): container finished" podID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" containerID="4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5" exitCode=0 Oct 30 20:18:29.410985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.410833 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" event={"ID":"48b890a8-5e32-4ddf-b802-0647ff7fe12a","Type":"ContainerDied","Data":"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5"} Oct 30 20:18:29.410985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.410916 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" event={"ID":"48b890a8-5e32-4ddf-b802-0647ff7fe12a","Type":"ContainerDied","Data":"0fe9ea0dfd98553768d36eca599fb783470c83039b395b68c35d7cf6d18bd38a"} Oct 30 20:18:29.410985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.410849 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl" Oct 30 20:18:29.411260 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.410951 4524 scope.go:122] "RemoveContainer" containerID="4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5" Oct 30 20:18:29.435039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.434426 4524 scope.go:122] "RemoveContainer" containerID="4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5" Oct 30 20:18:29.435039 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:29.434935 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5\": container with ID starting with 4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5 not found: ID does not exist" containerID="4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5" Oct 30 20:18:29.435039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.434973 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5"} err="failed to get container status \"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5\": rpc error: code = NotFound desc = could not find container \"4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5\": container with ID starting with 4fa143803541cda1e55fa9f95bb474c1d9e63c7bbc8b885bb86a824e70b3d7a5 not found: ID does not exist" Oct 30 20:18:29.671748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.665334 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:29.703003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.702966 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:18:29.706913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.706888 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3736ef9-6b7d9444c6-mm7nl"] Oct 30 20:18:29.924869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:29.921741 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:30.685154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:30.685116 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.023423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:31.023331 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.302891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:31.302814 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" path="/var/lib/kubelet/pods/48b890a8-5e32-4ddf-b802-0647ff7fe12a/volumes" Oct 30 20:18:31.380326 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:31.379993 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.771915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:31.771878 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:34.223041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.222981 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" podUID="a6332187-0427-4050-b378-430da06c18af" containerName="payload-processing" containerID="cri-o://dd8034f281417ec3a11e7de13304e1fc80ed09edd3d1a596577b0400e70da057" gracePeriod=25 Oct 30 20:18:34.322500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.322441 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" podUID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" containerName="payload-pre-processing" containerID="cri-o://2dcb755802dcef77f8e46f7223e67025447ed36e62dda5bba201a7a77002148f" gracePeriod=25 Oct 30 20:18:34.451452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.451159 4524 generic.go:356] "Generic (PLEG): container finished" podID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" containerID="2dcb755802dcef77f8e46f7223e67025447ed36e62dda5bba201a7a77002148f" exitCode=0 Oct 30 20:18:34.451452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.451325 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" event={"ID":"f338317f-bf2f-43d6-a8bc-2dfadebd4871","Type":"ContainerDied","Data":"2dcb755802dcef77f8e46f7223e67025447ed36e62dda5bba201a7a77002148f"} Oct 30 20:18:34.454388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.454276 4524 generic.go:356] "Generic (PLEG): container finished" podID="a6332187-0427-4050-b378-430da06c18af" containerID="dd8034f281417ec3a11e7de13304e1fc80ed09edd3d1a596577b0400e70da057" exitCode=0 Oct 30 20:18:34.454388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.454334 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" event={"ID":"a6332187-0427-4050-b378-430da06c18af","Type":"ContainerDied","Data":"dd8034f281417ec3a11e7de13304e1fc80ed09edd3d1a596577b0400e70da057"} Oct 30 20:18:34.539545 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.539517 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:18:34.654467 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.654429 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:18:34.658416 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.658390 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\" (UniqueName: \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\") pod \"a6332187-0427-4050-b378-430da06c18af\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " Oct 30 20:18:34.658561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.658495 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\") pod \"a6332187-0427-4050-b378-430da06c18af\" (UID: \"a6332187-0427-4050-b378-430da06c18af\") " Oct 30 20:18:34.658943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.658915 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume" pod "a6332187-0427-4050-b378-430da06c18af" (UID: "a6332187-0427-4050-b378-430da06c18af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:34.660629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.660600 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2" pod "a6332187-0427-4050-b378-430da06c18af" (UID: "a6332187-0427-4050-b378-430da06c18af"). InnerVolumeSpecName "kube-api-access-6p2n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:34.760064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.759985 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\" (UniqueName: \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\") pod \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " Oct 30 20:18:34.760289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.760072 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\") pod \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\" (UID: \"f338317f-bf2f-43d6-a8bc-2dfadebd4871\") " Oct 30 20:18:34.760496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.760465 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume" pod "f338317f-bf2f-43d6-a8bc-2dfadebd4871" (UID: "f338317f-bf2f-43d6-a8bc-2dfadebd4871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:34.760680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.760660 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f338317f-bf2f-43d6-a8bc-2dfadebd4871-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:34.760778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.760686 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p2n2\" (UniqueName: \"kubernetes.io/projected/a6332187-0427-4050-b378-430da06c18af-kube-api-access-6p2n2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:34.760778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.760731 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6332187-0427-4050-b378-430da06c18af-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:34.762277 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.762252 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn" pod "f338317f-bf2f-43d6-a8bc-2dfadebd4871" (UID: "f338317f-bf2f-43d6-a8bc-2dfadebd4871"). InnerVolumeSpecName "kube-api-access-drdtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:34.862098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:34.862058 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdtn\" (UniqueName: \"kubernetes.io/projected/f338317f-bf2f-43d6-a8bc-2dfadebd4871-kube-api-access-drdtn\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:35.463140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.462900 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" Oct 30 20:18:35.463140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.462930 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg" event={"ID":"f338317f-bf2f-43d6-a8bc-2dfadebd4871","Type":"ContainerDied","Data":"4f4d5a7f9cb38718203bada0d6a0c3551e215dd8832b3afe0926504ffaf8da6e"} Oct 30 20:18:35.463140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.463003 4524 scope.go:122] "RemoveContainer" containerID="2dcb755802dcef77f8e46f7223e67025447ed36e62dda5bba201a7a77002148f" Oct 30 20:18:35.467162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.467137 4524 generic.go:356] "Generic (PLEG): container finished" podID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerID="f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7" exitCode=0 Oct 30 20:18:35.467309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.467264 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerDied","Data":"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7"} Oct 30 20:18:35.467680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.467655 4524 scope.go:122] "RemoveContainer" containerID="f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7" Oct 30 20:18:35.469547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.469507 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:35.469658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.469592 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:35.469658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.469635 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:35.469972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.469947 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" Oct 30 20:18:35.470073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.469990 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb" event={"ID":"a6332187-0427-4050-b378-430da06c18af","Type":"ContainerDied","Data":"6c3a07378a3f63d1d0e879ab9729c70266cfddc7baec41653cf853a85abb3edd"} Oct 30 20:18:35.478312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.478290 4524 scope.go:122] "RemoveContainer" containerID="dd8034f281417ec3a11e7de13304e1fc80ed09edd3d1a596577b0400e70da057" Oct 30 20:18:35.570602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.570576 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:18:35.570842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.570818 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" podUID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" containerName="maas-api" containerID="cri-o://4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a" gracePeriod=30 Oct 30 20:18:35.633719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.633667 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:18:35.646724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.639963 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bb565e-66bbf7f847-7ztcg"] Oct 30 20:18:35.717755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.717668 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:35.749128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.749097 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:18:35.758527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.758306 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb565e-87d4d5f7d-d7tpb"] Oct 30 20:18:35.973780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:35.972640 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:18:36.073730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.066736 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\" (UniqueName: \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\") pod \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " Oct 30 20:18:36.073730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.067579 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\") pod \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " Oct 30 20:18:36.073730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.068157 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca" pod "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" (UID: "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:36.073730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.068371 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") pod \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " Oct 30 20:18:36.073730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.068461 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\") pod \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\" (UID: \"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9\") " Oct 30 20:18:36.082718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.080025 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:36.082718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.081014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls" pod "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" (UID: "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:36.085748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.083079 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb" pod "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" (UID: "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9"). InnerVolumeSpecName "kube-api-access-wnnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:36.085748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.084851 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs" pod "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" (UID: "44c2b553-1ba8-49cc-b53f-c73ff3d3fad9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:36.192104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.192068 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnvb\" (UniqueName: \"kubernetes.io/projected/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-kube-api-access-wnnvb\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:36.192104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.192105 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:36.192366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.192122 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:36.481122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.481086 4524 generic.go:356] "Generic (PLEG): container finished" podID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" containerID="4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a" exitCode=0 Oct 30 20:18:36.481605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.481141 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" Oct 30 20:18:36.481605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.481174 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" event={"ID":"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9","Type":"ContainerDied","Data":"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a"} Oct 30 20:18:36.481605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.481228 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx" event={"ID":"44c2b553-1ba8-49cc-b53f-c73ff3d3fad9","Type":"ContainerDied","Data":"7020d2fde8039ada9f5e70febb751977dc553021656c1ccc2f1e65bd5b0138fa"} Oct 30 20:18:36.481605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.481326 4524 scope.go:122] "RemoveContainer" containerID="4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a" Oct 30 20:18:36.485379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.485353 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerStarted","Data":"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5"} Oct 30 20:18:36.493096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.493075 4524 scope.go:122] "RemoveContainer" containerID="4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a" Oct 30 20:18:36.493400 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:36.493373 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a\": container with ID starting with 4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a not found: ID does not exist" containerID="4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a" Oct 30 20:18:36.493477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.493410 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a"} err="failed to get container status \"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a\": rpc error: code = NotFound desc = could not find container \"4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a\": container with ID starting with 4071c58958a7203f7ac885af3c3937f6bc1c21f53f1397030a0f32c805b4e69a not found: ID does not exist" Oct 30 20:18:36.672262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.672225 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:18:36.699776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.699729 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fcfa-7b997d4784-mnhjx"] Oct 30 20:18:36.967724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.964042 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:18:36.967724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.964159 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:36.967724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:36.964201 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:37.314083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.313994 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" path="/var/lib/kubelet/pods/44c2b553-1ba8-49cc-b53f-c73ff3d3fad9/volumes" Oct 30 20:18:37.314666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.314650 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6332187-0427-4050-b378-430da06c18af" path="/var/lib/kubelet/pods/a6332187-0427-4050-b378-430da06c18af/volumes" Oct 30 20:18:37.315043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.315029 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" path="/var/lib/kubelet/pods/f338317f-bf2f-43d6-a8bc-2dfadebd4871/volumes" Oct 30 20:18:37.818339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.818293 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:37.964723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.962014 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:37.964723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:37.962071 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:38.238045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:38.238015 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.575775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:38.568113 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.974112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:38.974074 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:38.974738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:38.974679 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:39.227293 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.227211 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.679268 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.679230 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.911528 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.911493 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.968790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.963052 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:39.968790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.963111 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:39.989822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.989073 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:18:39.990794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:39.990767 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" containerID="cri-o://a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b" gracePeriod=30 Oct 30 20:18:40.141597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:40.141558 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:40.971078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:40.971042 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:40.971349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:40.971318 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:41.078643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:41.078600 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.432004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:41.431971 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.962795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:41.962751 4524 patch_prober.go:29] interesting pod/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 20:18:41.963008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:41.962835 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 20:18:42.791812 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:42.791772 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:18:42.840275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:42.840244 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:42.968188 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:42.968156 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:18:42.968569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:42.968546 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:18:42.969336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:42.969317 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:18:43.086041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:43.085964 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:43.402375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:43.402341 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:45.408120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.408096 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:18:45.548036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.547999 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548051 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548104 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548164 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548203 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548266 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548309 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548336 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548391 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\") pod \"e9f9af97-35de-4af7-baf0-b53416f56e39\" (UID: \"e9f9af97-35de-4af7-baf0-b53416f56e39\") " Oct 30 20:18:45.548656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548468 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:45.548755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548393 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:45.548900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.548862 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:45.549024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.549003 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:45.549116 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.549070 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.549194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.549121 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.549194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.549136 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.549194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.549145 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:45.550938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.550907 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:45.551083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.550986 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:45.551421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.551399 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "kube-api-access-9t97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:45.551474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.551433 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token" pod "e9f9af97-35de-4af7-baf0-b53416f56e39" (UID: "e9f9af97-35de-4af7-baf0-b53416f56e39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:45.553751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.553664 4524 generic.go:356] "Generic (PLEG): container finished" podID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerID="a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b" exitCode=0 Oct 30 20:18:45.553751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.553729 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerDied","Data":"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b"} Oct 30 20:18:45.553883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.553747 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" Oct 30 20:18:45.553883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.553821 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6" event={"ID":"e9f9af97-35de-4af7-baf0-b53416f56e39","Type":"ContainerDied","Data":"4fb96801e31446d957e4098a16df62d25a4b9607105d74b689f37d1acf542548"} Oct 30 20:18:45.553883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.553852 4524 scope.go:122] "RemoveContainer" containerID="a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b" Oct 30 20:18:45.577341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.577323 4524 scope.go:122] "RemoveContainer" containerID="76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f" Oct 30 20:18:45.586331 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.586313 4524 scope.go:122] "RemoveContainer" containerID="a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b" Oct 30 20:18:45.586587 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:45.586569 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b\": container with ID starting with a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b not found: ID does not exist" containerID="a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b" Oct 30 20:18:45.586630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.586596 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b"} err="failed to get container status \"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b\": rpc error: code = NotFound desc = could not find container \"a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b\": container with ID starting with a1006ebcc9641b91032ae8044418f5095a0f07663da1a130e651376473e5782b not found: ID does not exist" Oct 30 20:18:45.586630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.586614 4524 scope.go:122] "RemoveContainer" containerID="76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f" Oct 30 20:18:45.586887 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:18:45.586868 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f\": container with ID starting with 76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f not found: ID does not exist" containerID="76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f" Oct 30 20:18:45.586954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.586899 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f"} err="failed to get container status \"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f\": rpc error: code = NotFound desc = could not find container \"76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f\": container with ID starting with 76be498bfe3d17f52045390cf80d7cb9586d7a6112fd5c95771675c81e678f1f not found: ID does not exist" Oct 30 20:18:45.650427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650396 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.650566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650431 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.650566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650446 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f9af97-35de-4af7-baf0-b53416f56e39-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.650566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650460 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t97t\" (UniqueName: \"kubernetes.io/projected/e9f9af97-35de-4af7-baf0-b53416f56e39-kube-api-access-9t97t\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.650566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650475 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.650566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.650488 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f9af97-35de-4af7-baf0-b53416f56e39-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:18:45.692861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.692832 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:18:45.697149 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:45.697127 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f3736ef9-openshift-default-5f9d95b86f-d4ll6"] Oct 30 20:18:46.090458 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:46.090420 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.380519 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:46.380444 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.758912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:46.758878 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:47.010984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:47.010911 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:47.300642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:47.300568 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" path="/var/lib/kubelet/pods/e9f9af97-35de-4af7-baf0-b53416f56e39/volumes" Oct 30 20:18:48.147352 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:48.147313 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:48.360023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:48.359994 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:48.737578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:48.737539 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:48.946934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:48.946907 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.216759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.216729 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:18:49.217455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217414 4524 cpu_manager.go:414] "containerMap: removing container" podUID="a6332187-0427-4050-b378-430da06c18af" containerName="payload-processing" Oct 30 20:18:49.217455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217439 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6332187-0427-4050-b378-430da06c18af" containerName="payload-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217476 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8640f882-eb60-4ee3-a0f1-2940dff09143" containerName="payload-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217486 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8640f882-eb60-4ee3-a0f1-2940dff09143" containerName="payload-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217532 4524 cpu_manager.go:414] "containerMap: removing container" podUID="da281092-bd97-41ef-a470-a47e285a7da4" containerName="payload-pre-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217540 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da281092-bd97-41ef-a470-a47e285a7da4" containerName="payload-pre-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217562 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217568 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217584 4524 cpu_manager.go:414] "containerMap: removing container" podUID="7e314883-d870-4fa3-9396-909f32ddae37" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217590 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e314883-d870-4fa3-9396-909f32ddae37" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217608 4524 cpu_manager.go:414] "containerMap: removing container" podUID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217615 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217633 4524 cpu_manager.go:414] "containerMap: removing container" podUID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217640 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" containerName="maas-api" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217663 4524 cpu_manager.go:414] "containerMap: removing container" podUID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" containerName="payload-pre-processing" Oct 30 20:18:49.217666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217670 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217682 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217689 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217869 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="da281092-bd97-41ef-a470-a47e285a7da4" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217886 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217898 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e314883-d870-4fa3-9396-909f32ddae37" containerName="maas-api" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217909 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b890a8-5e32-4ddf-b802-0647ff7fe12a" containerName="maas-api" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217920 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c2b553-1ba8-49cc-b53f-c73ff3d3fad9" containerName="maas-api" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217928 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217940 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="f338317f-bf2f-43d6-a8bc-2dfadebd4871" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217950 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8869a5-210e-4527-8c8c-85e72233665e" containerName="payload-pre-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217959 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6332187-0427-4050-b378-430da06c18af" containerName="payload-processing" Oct 30 20:18:49.218373 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.217968 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8640f882-eb60-4ee3-a0f1-2940dff09143" containerName="payload-processing" Oct 30 20:18:49.223435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.223413 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.325842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.325799 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.326219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.326186 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.326448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.326417 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.326863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.326840 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.327024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.327006 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.327137 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.327124 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.327294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.327280 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.327390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.327377 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.327483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.327469 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghlb2\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.365141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.365102 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0051fb-openshift-default-dockercfg-pg969\"" Oct 30 20:18:49.373921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.373893 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:18:49.425617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.423176 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.430301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.430263 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.430533 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.430511 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.430666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.430645 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghlb2\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.430892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.430863 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.431121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.431096 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.431241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.431226 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.431398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.431383 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.431540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.431516 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.431662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.431646 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.445724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.441621 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.469760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.457097 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.469760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.457712 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.469760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.460606 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.469760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.461918 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.482891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.478401 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.485324 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.485291 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.527186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.510239 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.530280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.530248 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghlb2\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\") pod \"e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:49.665292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.665255 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.684446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:49.684419 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:50.122579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.122548 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:18:50.124076 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:18:50.124052 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8286b115_c193_457f_8882_bf43b1fe4b3c.slice/crio-643554991b142fdc9e0f9fe9e7b2799e151371e0925c2c9beecf9e20a85c9186 WatchSource:0}: Error finding container 643554991b142fdc9e0f9fe9e7b2799e151371e0925c2c9beecf9e20a85c9186: Status 404 returned error can't find the container with id 643554991b142fdc9e0f9fe9e7b2799e151371e0925c2c9beecf9e20a85c9186 Oct 30 20:18:50.126855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.126825 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:50.126970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.126906 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:50.126970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.126954 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:50.626254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.626211 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerStarted","Data":"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5"} Oct 30 20:18:50.626669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.626282 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerStarted","Data":"643554991b142fdc9e0f9fe9e7b2799e151371e0925c2c9beecf9e20a85c9186"} Oct 30 20:18:50.684810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.684777 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:50.751378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:50.751314 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podStartSLOduration=1.751293348 podStartE2EDuration="1.751293348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:50.747388755 +0000 UTC m=+2225.591060385" watchObservedRunningTime="2026-10-30 20:18:50.751293348 +0000 UTC m=+2225.594964978" Oct 30 20:18:51.690888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:51.690860 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:52.639469 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:52.639442 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:52.640480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:52.640460 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:18:53.837994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:53.837960 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:54.145317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:54.145285 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:54.398004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:54.397914 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:55.821825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:18:55.821793 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:00.387741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:00.387687 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.325753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.325716 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.567726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.565424 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.668385 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.668350 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:19:02.669021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.669006 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:19:02.669082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.669022 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f9af97-35de-4af7-baf0-b53416f56e39" containerName="istio-proxy" Oct 30 20:19:02.672475 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.672458 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.717535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.717507 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrnv\" (UniqueName: \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.717658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.717593 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.778726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.777104 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0051fb\"" Oct 30 20:19:02.778726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.777223 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0051fb-dockercfg-tjlfk\"" Oct 30 20:19:02.807069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.807034 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:19:02.839723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.825771 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrnv\" (UniqueName: \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.840167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.840124 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.843672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.841512 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.949461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.942362 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrnv\" (UniqueName: \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\") pod \"payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:02.949710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.944186 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:19:02.954518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.954497 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:19:02.954651 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.954629 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:02.988665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:02.988636 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:03.046427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.046401 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.046533 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.046471 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95qp\" (UniqueName: \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.148428 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.148383 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.148681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.148462 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95qp\" (UniqueName: \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.149180 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.149154 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.211502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.211434 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95qp\" (UniqueName: \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\") pod \"payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.274524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.274491 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:03.372350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.372303 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:19:03.423888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.423855 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:03.659181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.659157 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:19:03.660908 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:03.660876 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd70019_8732_45c7_a432_9d78df73b7ba.slice/crio-6b0d85889e6b6991bc6e791fc8548fba6482a83821026e302f27791fc989c8f1 WatchSource:0}: Error finding container 6b0d85889e6b6991bc6e791fc8548fba6482a83821026e302f27791fc989c8f1: Status 404 returned error can't find the container with id 6b0d85889e6b6991bc6e791fc8548fba6482a83821026e302f27791fc989c8f1 Oct 30 20:19:03.708279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.708248 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" event={"ID":"b3870242-9d90-4662-8a01-d06fe5defbc1","Type":"ContainerStarted","Data":"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb"} Oct 30 20:19:03.708409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.708309 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" event={"ID":"b3870242-9d90-4662-8a01-d06fe5defbc1","Type":"ContainerStarted","Data":"a0b66a0df58ed7457011b38214f4db5b0bc8112d20f245dbae2ed3c7449822a4"} Oct 30 20:19:03.708409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.708365 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:03.709637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.709611 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" event={"ID":"4bd70019-8732-45c7-a432-9d78df73b7ba","Type":"ContainerStarted","Data":"6b0d85889e6b6991bc6e791fc8548fba6482a83821026e302f27791fc989c8f1"} Oct 30 20:19:03.837649 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:03.837603 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" podStartSLOduration=1.837589616 podStartE2EDuration="1.837589616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:03.834822931 +0000 UTC m=+2238.678494561" watchObservedRunningTime="2026-10-30 20:19:03.837589616 +0000 UTC m=+2238.681261245" Oct 30 20:19:04.481430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:04.481341 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:04.717582 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:04.717551 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" event={"ID":"4bd70019-8732-45c7-a432-9d78df73b7ba","Type":"ContainerStarted","Data":"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104"} Oct 30 20:19:04.717915 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:04.717899 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:04.834358 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:04.834256 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" podStartSLOduration=2.83424136 podStartE2EDuration="2.83424136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:04.833390365 +0000 UTC m=+2239.677062017" watchObservedRunningTime="2026-10-30 20:19:04.83424136 +0000 UTC m=+2239.677912980" Oct 30 20:19:09.415238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:09.415200 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:10.016500 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:10.016468 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:19:10.459252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:10.459215 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:10.732273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:10.732190 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:12.787654 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:12.787612 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:19:14.240823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:14.240781 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:14.721282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:14.721257 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:19:15.020191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.019927 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" podUID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" containerName="payload-processing" containerID="cri-o://c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4" gracePeriod=25 Oct 30 20:19:15.306397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.306359 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:19:15.369280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.369243 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\") pod \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " Oct 30 20:19:15.369420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.369311 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\" (UniqueName: \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\") pod \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\" (UID: \"b656d072-a2a0-4021-b863-0d5e2ec2b3b5\") " Oct 30 20:19:15.369794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.369759 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume" pod "b656d072-a2a0-4021-b863-0d5e2ec2b3b5" (UID: "b656d072-a2a0-4021-b863-0d5e2ec2b3b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:15.370063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.370036 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:15.371644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.371618 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz" pod "b656d072-a2a0-4021-b863-0d5e2ec2b3b5" (UID: "b656d072-a2a0-4021-b863-0d5e2ec2b3b5"). InnerVolumeSpecName "kube-api-access-qgnrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:15.475759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.474369 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnrz\" (UniqueName: \"kubernetes.io/projected/b656d072-a2a0-4021-b863-0d5e2ec2b3b5-kube-api-access-qgnrz\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:15.726388 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.726359 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:19:15.812988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.812958 4524 generic.go:356] "Generic (PLEG): container finished" podID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" containerID="c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4" exitCode=0 Oct 30 20:19:15.813165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.813005 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" Oct 30 20:19:15.813165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.813025 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" event={"ID":"b656d072-a2a0-4021-b863-0d5e2ec2b3b5","Type":"ContainerDied","Data":"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4"} Oct 30 20:19:15.813165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.813064 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j" event={"ID":"b656d072-a2a0-4021-b863-0d5e2ec2b3b5","Type":"ContainerDied","Data":"b2e1d72e8722de207abf6e2b804d078afa334fed27df8821b19f9a77b51b511b"} Oct 30 20:19:15.813165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.813087 4524 scope.go:122] "RemoveContainer" containerID="c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4" Oct 30 20:19:15.823182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.823156 4524 scope.go:122] "RemoveContainer" containerID="c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4" Oct 30 20:19:15.823463 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:15.823440 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4\": container with ID starting with c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4 not found: ID does not exist" containerID="c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4" Oct 30 20:19:15.823518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.823469 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4"} err="failed to get container status \"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4\": rpc error: code = NotFound desc = could not find container \"c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4\": container with ID starting with c2d1904c317fe572b80b4ff1c39e958e617aea96aeecbb07c2a16701ec7a5dd4 not found: ID does not exist" Oct 30 20:19:15.967201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.967159 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:19:15.970172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:15.970145 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f44ec2-84f4c976bd-ftv2j"] Oct 30 20:19:17.307842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:17.307806 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" path="/var/lib/kubelet/pods/b656d072-a2a0-4021-b863-0d5e2ec2b3b5/volumes" Oct 30 20:19:18.371619 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.371582 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:19:18.372124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.371848 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" podUID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" containerName="maas-api" containerID="cri-o://a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950" gracePeriod=30 Oct 30 20:19:18.533577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.533547 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:18.709033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.709000 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:19:18.818391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.817146 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\") pod \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " Oct 30 20:19:18.818391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.817271 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\") pod \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " Oct 30 20:19:18.818391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.817395 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\") pod \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " Oct 30 20:19:18.818391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.817520 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\" (UniqueName: \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\") pod \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\" (UID: \"f6338bd1-4f84-4849-bea8-f8c04e6966ea\") " Oct 30 20:19:18.819821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.819726 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca" pod "f6338bd1-4f84-4849-bea8-f8c04e6966ea" (UID: "f6338bd1-4f84-4849-bea8-f8c04e6966ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:18.825137 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.825058 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs" pod "f6338bd1-4f84-4849-bea8-f8c04e6966ea" (UID: "f6338bd1-4f84-4849-bea8-f8c04e6966ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:18.829073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.829028 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls" pod "f6338bd1-4f84-4849-bea8-f8c04e6966ea" (UID: "f6338bd1-4f84-4849-bea8-f8c04e6966ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:18.834091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.833947 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8" pod "f6338bd1-4f84-4849-bea8-f8c04e6966ea" (UID: "f6338bd1-4f84-4849-bea8-f8c04e6966ea"). InnerVolumeSpecName "kube-api-access-fkvj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:18.847724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.847681 4524 generic.go:356] "Generic (PLEG): container finished" podID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" containerID="a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950" exitCode=0 Oct 30 20:19:18.848012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.847861 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" event={"ID":"f6338bd1-4f84-4849-bea8-f8c04e6966ea","Type":"ContainerDied","Data":"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950"} Oct 30 20:19:18.848106 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.847938 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" Oct 30 20:19:18.856310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.856279 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb" event={"ID":"f6338bd1-4f84-4849-bea8-f8c04e6966ea","Type":"ContainerDied","Data":"9a28aac72e6ee05641546ee23943da52f832427138a42cf547e6ffbe5b7daa34"} Oct 30 20:19:18.856447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.856353 4524 scope.go:122] "RemoveContainer" containerID="a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950" Oct 30 20:19:18.887314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.887285 4524 scope.go:122] "RemoveContainer" containerID="a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950" Oct 30 20:19:18.888474 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:18.888432 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950\": container with ID starting with a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950 not found: ID does not exist" containerID="a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950" Oct 30 20:19:18.888583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.888485 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950"} err="failed to get container status \"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950\": rpc error: code = NotFound desc = could not find container \"a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950\": container with ID starting with a5c9ed23903ea5ea2d16af0da7356b03b8865f86ff8e73091809b6ba6c020950 not found: ID does not exist" Oct 30 20:19:18.918977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.918955 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkvj8\" (UniqueName: \"kubernetes.io/projected/f6338bd1-4f84-4849-bea8-f8c04e6966ea-kube-api-access-fkvj8\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:18.919070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.918979 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:18.919070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.918989 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6338bd1-4f84-4849-bea8-f8c04e6966ea-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:18.919070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:18.918998 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6338bd1-4f84-4849-bea8-f8c04e6966ea-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:19.027789 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.027535 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:19:19.034839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.034807 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d2c70-68cf866449-x8zrb"] Oct 30 20:19:19.304094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.303848 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" path="/var/lib/kubelet/pods/f6338bd1-4f84-4849-bea8-f8c04e6966ea/volumes" Oct 30 20:19:19.543312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.543279 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:19:19.543713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.543475 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" podUID="49ea0538-d11e-4806-a36e-0a1cb1253225" containerName="maas-api" containerID="cri-o://8aee1b6a0c994f8b5ca0062cd4d804c19cb2cf892acd1746e3d35a882540ebe0" gracePeriod=30 Oct 30 20:19:19.706236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.706199 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:19.870329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.870161 4524 generic.go:356] "Generic (PLEG): container finished" podID="49ea0538-d11e-4806-a36e-0a1cb1253225" containerID="8aee1b6a0c994f8b5ca0062cd4d804c19cb2cf892acd1746e3d35a882540ebe0" exitCode=0 Oct 30 20:19:19.870329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.870249 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" event={"ID":"49ea0538-d11e-4806-a36e-0a1cb1253225","Type":"ContainerDied","Data":"8aee1b6a0c994f8b5ca0062cd4d804c19cb2cf892acd1746e3d35a882540ebe0"} Oct 30 20:19:19.905266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.905241 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:19:19.948528 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.948492 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\") pod \"49ea0538-d11e-4806-a36e-0a1cb1253225\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " Oct 30 20:19:19.948711 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.948613 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\") pod \"49ea0538-d11e-4806-a36e-0a1cb1253225\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " Oct 30 20:19:19.948766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.948755 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") pod \"49ea0538-d11e-4806-a36e-0a1cb1253225\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " Oct 30 20:19:19.948874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.948848 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca" pod "49ea0538-d11e-4806-a36e-0a1cb1253225" (UID: "49ea0538-d11e-4806-a36e-0a1cb1253225"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:19.948922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.948881 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\" (UniqueName: \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\") pod \"49ea0538-d11e-4806-a36e-0a1cb1253225\" (UID: \"49ea0538-d11e-4806-a36e-0a1cb1253225\") " Oct 30 20:19:19.949392 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.949365 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea0538-d11e-4806-a36e-0a1cb1253225-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:19.951486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.951014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs" pod "49ea0538-d11e-4806-a36e-0a1cb1253225" (UID: "49ea0538-d11e-4806-a36e-0a1cb1253225"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:19.951486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.951411 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls" pod "49ea0538-d11e-4806-a36e-0a1cb1253225" (UID: "49ea0538-d11e-4806-a36e-0a1cb1253225"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:19.951486 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:19.951443 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v" pod "49ea0538-d11e-4806-a36e-0a1cb1253225" (UID: "49ea0538-d11e-4806-a36e-0a1cb1253225"). InnerVolumeSpecName "kube-api-access-b6v7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:20.050711 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.050663 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:20.050711 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.050694 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v7v\" (UniqueName: \"kubernetes.io/projected/49ea0538-d11e-4806-a36e-0a1cb1253225-kube-api-access-b6v7v\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:20.050711 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.050720 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ea0538-d11e-4806-a36e-0a1cb1253225-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:20.167391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.159385 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:20.401105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.400381 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:20.881823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.881796 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" Oct 30 20:19:20.882282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.881818 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt" event={"ID":"49ea0538-d11e-4806-a36e-0a1cb1253225","Type":"ContainerDied","Data":"afcf7fc5144451d4a9a57e5a819d868d354556632f62388c98e8245d1a082655"} Oct 30 20:19:20.882282 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:20.881893 4524 scope.go:122] "RemoveContainer" containerID="8aee1b6a0c994f8b5ca0062cd4d804c19cb2cf892acd1746e3d35a882540ebe0" Oct 30 20:19:21.047944 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.047907 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:19:21.055200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.055171 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-66fpt"] Oct 30 20:19:21.075503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.075473 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:21.302182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.302150 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ea0538-d11e-4806-a36e-0a1cb1253225" path="/var/lib/kubelet/pods/49ea0538-d11e-4806-a36e-0a1cb1253225/volumes" Oct 30 20:19:21.382415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.382373 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:21.725305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.725216 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:21.944510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:21.944477 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:26.285691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:26.285652 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:26.998879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:26.998805 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.578158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:28.578112 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.824792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:28.824760 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:29.091471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:29.091426 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:30.033657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:30.033618 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:30.250332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:30.250299 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.399431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.399399 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.649768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.641826 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.861360 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.861324 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.885581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.885546 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:19:32.886245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886227 4524 cpu_manager.go:414] "containerMap: removing container" podUID="49ea0538-d11e-4806-a36e-0a1cb1253225" containerName="maas-api" Oct 30 20:19:32.886245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886245 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea0538-d11e-4806-a36e-0a1cb1253225" containerName="maas-api" Oct 30 20:19:32.886366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886261 4524 cpu_manager.go:414] "containerMap: removing container" podUID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" containerName="payload-processing" Oct 30 20:19:32.886366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886267 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" containerName="payload-processing" Oct 30 20:19:32.886366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886279 4524 cpu_manager.go:414] "containerMap: removing container" podUID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" containerName="maas-api" Oct 30 20:19:32.886366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886284 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" containerName="maas-api" Oct 30 20:19:32.886366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886364 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="b656d072-a2a0-4021-b863-0d5e2ec2b3b5" containerName="payload-processing" Oct 30 20:19:32.886520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886375 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6338bd1-4f84-4849-bea8-f8c04e6966ea" containerName="maas-api" Oct 30 20:19:32.886520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.886386 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea0538-d11e-4806-a36e-0a1cb1253225" containerName="maas-api" Oct 30 20:19:32.891325 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.891307 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:32.975309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.975229 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwxg\" (UniqueName: \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:32.975309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:32.975286 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.044555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.005613 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-37732de7\"" Oct 30 20:19:33.052055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.051978 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-37732de7-dockercfg-k4xq2\"" Oct 30 20:19:33.059056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.059025 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:19:33.087822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.087772 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwxg\" (UniqueName: \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.088019 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.087836 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.091752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.088871 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.091752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.088932 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:19:33.095974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.095948 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.101312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.098914 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:19:33.169520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.169488 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwxg\" (UniqueName: \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\") pod \"payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.188976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.188942 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84ck\" (UniqueName: \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.189132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.189103 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.205627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.205600 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.291606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.291570 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.291835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.291810 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84ck\" (UniqueName: \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.292547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.292497 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.353725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.353675 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84ck\" (UniqueName: \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.422856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.422828 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:33.585969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.585944 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:19:33.587756 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:33.587723 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bcec45_1480_41cd_aca1_ca4705351e92.slice/crio-56482691f3ceba5951ddcaf7f427a1998893ff0b7663f81c13ce5dc2bcd8d2a5 WatchSource:0}: Error finding container 56482691f3ceba5951ddcaf7f427a1998893ff0b7663f81c13ce5dc2bcd8d2a5: Status 404 returned error can't find the container with id 56482691f3ceba5951ddcaf7f427a1998893ff0b7663f81c13ce5dc2bcd8d2a5 Oct 30 20:19:33.801673 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:33.801643 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafc2647_ac35_4bb3_a0ff_6ebc1cba326f.slice/crio-4a5dc2196b0bf939dfeaba843ba4a7d00e2172ee1eb747fec02e239999da422b WatchSource:0}: Error finding container 4a5dc2196b0bf939dfeaba843ba4a7d00e2172ee1eb747fec02e239999da422b: Status 404 returned error can't find the container with id 4a5dc2196b0bf939dfeaba843ba4a7d00e2172ee1eb747fec02e239999da422b Oct 30 20:19:33.801856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.801713 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:19:33.966962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.966918 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" event={"ID":"49bcec45-1480-41cd-aca1-ca4705351e92","Type":"ContainerStarted","Data":"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6"} Oct 30 20:19:33.967152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.967023 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:33.967152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.967051 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" event={"ID":"49bcec45-1480-41cd-aca1-ca4705351e92","Type":"ContainerStarted","Data":"56482691f3ceba5951ddcaf7f427a1998893ff0b7663f81c13ce5dc2bcd8d2a5"} Oct 30 20:19:33.969186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.969154 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" event={"ID":"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f","Type":"ContainerStarted","Data":"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009"} Oct 30 20:19:33.969313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.969200 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" event={"ID":"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f","Type":"ContainerStarted","Data":"4a5dc2196b0bf939dfeaba843ba4a7d00e2172ee1eb747fec02e239999da422b"} Oct 30 20:19:33.969313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:33.969247 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:34.089070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:34.088980 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" podStartSLOduration=2.088966964 podStartE2EDuration="2.088966964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:34.087006122 +0000 UTC m=+2268.930677751" watchObservedRunningTime="2026-10-30 20:19:34.088966964 +0000 UTC m=+2268.932638592" Oct 30 20:19:34.231492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:34.226581 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" podStartSLOduration=2.226560221 podStartE2EDuration="2.226560221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:34.221143782 +0000 UTC m=+2269.064815413" watchObservedRunningTime="2026-10-30 20:19:34.226560221 +0000 UTC m=+2269.070231851" Oct 30 20:19:36.676580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:36.676547 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:19:36.676978 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:36.676779 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" podUID="8c957d15-c365-4aae-a8de-b34d0ec66635" containerName="maas-api" containerID="cri-o://ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e" gracePeriod=30 Oct 30 20:19:36.894725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:36.894322 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:36.991838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:36.991813 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:19:37.000384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.000354 4524 generic.go:356] "Generic (PLEG): container finished" podID="8c957d15-c365-4aae-a8de-b34d0ec66635" containerID="ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e" exitCode=0 Oct 30 20:19:37.000508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.000391 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" event={"ID":"8c957d15-c365-4aae-a8de-b34d0ec66635","Type":"ContainerDied","Data":"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e"} Oct 30 20:19:37.000508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.000429 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" event={"ID":"8c957d15-c365-4aae-a8de-b34d0ec66635","Type":"ContainerDied","Data":"d08e710a2a57ef66376f6c4fd37e0b30f52f527e95b9ac40a54605593803352d"} Oct 30 20:19:37.000508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.000455 4524 scope.go:122] "RemoveContainer" containerID="ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e" Oct 30 20:19:37.000508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.000395 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p" Oct 30 20:19:37.010073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.010047 4524 scope.go:122] "RemoveContainer" containerID="ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e" Oct 30 20:19:37.010396 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:37.010376 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e\": container with ID starting with ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e not found: ID does not exist" containerID="ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e" Oct 30 20:19:37.010478 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.010405 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e"} err="failed to get container status \"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e\": rpc error: code = NotFound desc = could not find container \"ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e\": container with ID starting with ecbcaede36c4b487a5e1a404acdf6b3e5f855429a99acb2d0fdf14bb783cf59e not found: ID does not exist" Oct 30 20:19:37.178376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.178340 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\") pod \"8c957d15-c365-4aae-a8de-b34d0ec66635\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " Oct 30 20:19:37.178579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.178408 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\" (UniqueName: \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\") pod \"8c957d15-c365-4aae-a8de-b34d0ec66635\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " Oct 30 20:19:37.178579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.178459 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\") pod \"8c957d15-c365-4aae-a8de-b34d0ec66635\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " Oct 30 20:19:37.178779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.178751 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") pod \"8c957d15-c365-4aae-a8de-b34d0ec66635\" (UID: \"8c957d15-c365-4aae-a8de-b34d0ec66635\") " Oct 30 20:19:37.179082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.179051 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca" pod "8c957d15-c365-4aae-a8de-b34d0ec66635" (UID: "8c957d15-c365-4aae-a8de-b34d0ec66635"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:37.179620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.179591 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c957d15-c365-4aae-a8de-b34d0ec66635-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:37.181474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.181446 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2" pod "8c957d15-c365-4aae-a8de-b34d0ec66635" (UID: "8c957d15-c365-4aae-a8de-b34d0ec66635"). InnerVolumeSpecName "kube-api-access-7dls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:37.181647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.181620 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs" pod "8c957d15-c365-4aae-a8de-b34d0ec66635" (UID: "8c957d15-c365-4aae-a8de-b34d0ec66635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:37.181852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.181828 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls" pod "8c957d15-c365-4aae-a8de-b34d0ec66635" (UID: "8c957d15-c365-4aae-a8de-b34d0ec66635"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:37.287791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.280834 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:37.287791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.280870 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c957d15-c365-4aae-a8de-b34d0ec66635-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:37.287791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.280886 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dls2\" (UniqueName: \"kubernetes.io/projected/8c957d15-c365-4aae-a8de-b34d0ec66635-kube-api-access-7dls2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:37.428284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.428247 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:19:37.432776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:37.432747 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bb565e-76ffbfb9db-nx79p"] Oct 30 20:19:38.430094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:38.430061 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:38.656956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:38.656921 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:38.867153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:38.867070 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:39.301620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:39.301588 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c957d15-c365-4aae-a8de-b34d0ec66635" path="/var/lib/kubelet/pods/8c957d15-c365-4aae-a8de-b34d0ec66635/volumes" Oct 30 20:19:39.437281 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:39.437245 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:41.410686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.410654 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:19:41.411245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.411230 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8c957d15-c365-4aae-a8de-b34d0ec66635" containerName="maas-api" Oct 30 20:19:41.411292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.411246 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c957d15-c365-4aae-a8de-b34d0ec66635" containerName="maas-api" Oct 30 20:19:41.411351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.411341 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c957d15-c365-4aae-a8de-b34d0ec66635" containerName="maas-api" Oct 30 20:19:41.414836 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.414816 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439240 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439349 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439408 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439436 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439503 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjnj\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439613 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439783 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439766 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439799 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.439896 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.439859 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.460401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.460380 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:19:41.464317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.464298 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:19:41.464423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.464403 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.468182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.468166 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.476471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.476452 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:19:41.480379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.480360 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.507991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.507968 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\"" Oct 30 20:19:41.518174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.518153 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:19:41.541124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541100 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.541238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541136 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.541238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541162 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541269 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.541357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541317 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.541513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541395 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541439 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.541513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541452 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541459 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v6k\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.541513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541509 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541539 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541582 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541607 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541650 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjnj\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541678 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541761 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.541806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541792 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541904 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.541938 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgzc\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542014 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfznj\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542039 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.542084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542050 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542100 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542118 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542175 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542217 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542242 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542279 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542302 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542324 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542352 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542445 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542498 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542516 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542555 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.542916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542575 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.542916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542590 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.542916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542618 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.542916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542637 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.542916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.542887 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.543081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.543028 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.543853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.543836 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.544890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.544863 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.569200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.569167 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:19:41.575119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.575092 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:19:41.604286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.604251 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.605349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.605323 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjnj\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-q28fg\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.635336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.635306 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:19:41.644127 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644103 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.644254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644149 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v6k\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.644254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644192 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644237 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644265 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644317 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644353 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644379 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgzc\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644422 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfznj\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644446 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.644781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644471 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.644781 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644516 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.644898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644518 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.644898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644858 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644899 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644936 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.644976 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645016 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645067 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645101 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645131 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645152 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.645175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645165 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645199 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645232 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645268 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645306 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645352 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645387 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.645495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645451 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.645893 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.645736 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.646196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.646175 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.648121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648089 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.648488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648469 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.648665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648650 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.648782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648717 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.648852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648801 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.648852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.648845 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.649085 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.649060 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.649684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.649659 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.649991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.649968 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.650565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.650546 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.651231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.651212 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.651441 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.651423 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.651640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.651622 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.652052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.652025 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.652525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.652003 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.654904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.654873 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.706241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.706173 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v6k\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.707622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.707599 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.707731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.707684 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.711611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.711590 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgzc\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-v95qm\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.711738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.711627 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfznj\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t624r\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:41.711885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.711868 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2z4mx\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.728410 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.728391 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:41.779048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.778864 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:41.788019 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.787986 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:41.800206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:41.800176 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:42.161365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.161339 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:19:42.162162 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:42.162139 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321a585b_81da_4d31_af5b_4de25af3e1fb.slice/crio-cb1311f2c10d20467d630dbd1d1ee684a4d889bd1b95a6af56c3440240a6a4df WatchSource:0}: Error finding container cb1311f2c10d20467d630dbd1d1ee684a4d889bd1b95a6af56c3440240a6a4df: Status 404 returned error can't find the container with id cb1311f2c10d20467d630dbd1d1ee684a4d889bd1b95a6af56c3440240a6a4df Oct 30 20:19:42.164714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.164664 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.164834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.164762 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.164834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.164807 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.217255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.217229 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:19:42.218311 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:42.218288 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c1c650_6d4c_45cb_b5f2_56e523b54365.slice/crio-cac623b62ea61a76a148d82da1d34e0ca47eea0aaaa9465302fded43b5a22cea WatchSource:0}: Error finding container cac623b62ea61a76a148d82da1d34e0ca47eea0aaaa9465302fded43b5a22cea: Status 404 returned error can't find the container with id cac623b62ea61a76a148d82da1d34e0ca47eea0aaaa9465302fded43b5a22cea Oct 30 20:19:42.221212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.221177 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.221334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.221279 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.221334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.221317 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.265746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.265309 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:19:42.269039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.268969 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:19:42.269039 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:42.269002 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e262ae1_982f_4de1_9f21_422fa840d6bd.slice/crio-6dce1f0d87e81ac829ac08c7020d5044ad381567814c861997bc0745c9cdf190 WatchSource:0}: Error finding container 6dce1f0d87e81ac829ac08c7020d5044ad381567814c861997bc0745c9cdf190: Status 404 returned error can't find the container with id 6dce1f0d87e81ac829ac08c7020d5044ad381567814c861997bc0745c9cdf190 Oct 30 20:19:42.272058 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:19:42.271019 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871b5b61_8f4a_4e16_bff7_f18801b67b21.slice/crio-a82f8ec114aa49c6dd9cf88c69b35a008dd6876c9edba5ed1ecaa3491e5e3760 WatchSource:0}: Error finding container a82f8ec114aa49c6dd9cf88c69b35a008dd6876c9edba5ed1ecaa3491e5e3760: Status 404 returned error can't find the container with id a82f8ec114aa49c6dd9cf88c69b35a008dd6876c9edba5ed1ecaa3491e5e3760 Oct 30 20:19:42.272492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.272454 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.272598 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.272543 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.272598 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.272591 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.273728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.273622 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.273728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.273676 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.273897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:42.273756 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:42.800606 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:42.800536 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:19:43.116753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.116630 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" event={"ID":"1e262ae1-982f-4de1-9f21-422fa840d6bd","Type":"ContainerStarted","Data":"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6"} Oct 30 20:19:43.117047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.117017 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" event={"ID":"1e262ae1-982f-4de1-9f21-422fa840d6bd","Type":"ContainerStarted","Data":"6dce1f0d87e81ac829ac08c7020d5044ad381567814c861997bc0745c9cdf190"} Oct 30 20:19:43.169724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.147067 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" event={"ID":"f9c1c650-6d4c-45cb-b5f2-56e523b54365","Type":"ContainerStarted","Data":"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe"} Oct 30 20:19:43.169724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.147138 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" event={"ID":"f9c1c650-6d4c-45cb-b5f2-56e523b54365","Type":"ContainerStarted","Data":"cac623b62ea61a76a148d82da1d34e0ca47eea0aaaa9465302fded43b5a22cea"} Oct 30 20:19:43.169724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.166874 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" event={"ID":"871b5b61-8f4a-4e16-bff7-f18801b67b21","Type":"ContainerStarted","Data":"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d"} Oct 30 20:19:43.169724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.166953 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" event={"ID":"871b5b61-8f4a-4e16-bff7-f18801b67b21","Type":"ContainerStarted","Data":"a82f8ec114aa49c6dd9cf88c69b35a008dd6876c9edba5ed1ecaa3491e5e3760"} Oct 30 20:19:43.187599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.186845 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" event={"ID":"321a585b-81da-4d31-af5b-4de25af3e1fb","Type":"ContainerStarted","Data":"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c"} Oct 30 20:19:43.187599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.186909 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" event={"ID":"321a585b-81da-4d31-af5b-4de25af3e1fb","Type":"ContainerStarted","Data":"cb1311f2c10d20467d630dbd1d1ee684a4d889bd1b95a6af56c3440240a6a4df"} Oct 30 20:19:43.282669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.282592 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podStartSLOduration=2.282567575 podStartE2EDuration="2.282567575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:43.256998529 +0000 UTC m=+2278.100670161" watchObservedRunningTime="2026-10-30 20:19:43.282567575 +0000 UTC m=+2278.126239206" Oct 30 20:19:43.721724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.721388 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podStartSLOduration=2.721363658 podStartE2EDuration="2.721363658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:43.598138927 +0000 UTC m=+2278.441810556" watchObservedRunningTime="2026-10-30 20:19:43.721363658 +0000 UTC m=+2278.565035289" Oct 30 20:19:43.759063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.759028 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:43.761313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.761274 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:43.761561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.761526 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:43.790193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.790157 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:43.795733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.794836 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:43.805442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.805396 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:43.812693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.806018 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:43.825238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.807620 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:43.825450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.806095 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:43.825592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.825564 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:43.827604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.827575 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:43.838935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.827763 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:43.898723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.884517 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podStartSLOduration=2.884496369 podStartE2EDuration="2.884496369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:43.756057364 +0000 UTC m=+2278.599729015" watchObservedRunningTime="2026-10-30 20:19:43.884496369 +0000 UTC m=+2278.728167999" Oct 30 20:19:43.898723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.888643 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:43.912186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:43.912118 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podStartSLOduration=2.912099838 podStartE2EDuration="2.912099838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:43.888334267 +0000 UTC m=+2278.732005897" watchObservedRunningTime="2026-10-30 20:19:43.912099838 +0000 UTC m=+2278.755771470" Oct 30 20:19:44.185322 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.175680 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:44.479921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.479804 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:44.602653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.602617 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:44.620729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.617017 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:44.734539 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.734425 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:44.734846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.734811 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:44.782295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.782245 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:44.782621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.782573 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:44.792727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.792675 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:44.792964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.792922 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:44.793216 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.793135 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:44.818655 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.818622 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:44.819273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.819225 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:44.876862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.876821 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 20:19:44.947876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:44.947830 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:19:45.017024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.016915 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:19:45.018843 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.018820 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:19:45.636753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.622646 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:45.642190 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.642151 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:45.760753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.743785 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:45.760753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.743843 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:45.764011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.763979 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:19:45.787042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.787008 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:45.820128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.820091 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:45.820648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.795775 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:45.821095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.821074 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:19:45.828732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.819007 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:45.828732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.824348 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:45.828732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.819169 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:45.828732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.824396 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:45.976117 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.976072 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:46.005725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:45.982658 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:19:46.437737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.434166 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:19:46.624191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.624138 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:46.745914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.745854 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:46.746831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.746809 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:19:46.781729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.773119 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:46.781964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.781931 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.858015 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.858075 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.858165 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.858187 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.859505 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:46.859751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.859546 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:47.000017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.957046 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:19:47.000017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:46.957198 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:19:47.205162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.193948 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:47.359777 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.359491 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:47.434833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.434787 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:47.667724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.666792 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:47.771724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.770949 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:47.771724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.771026 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:47.846722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.827814 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:47.846722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.827881 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:47.932780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.900104 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:47.953408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.953365 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:47.989718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:47.954851 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:19:48.086723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.027918 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.086723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.028013 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:48.172727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.093301 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.172727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.099080 4524 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:48.172727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.099297 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:19:48.264779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.207838 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:48.473757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.463778 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:19:48.473757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.464159 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" containerID="cri-o://04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0" gracePeriod=30 Oct 30 20:19:48.488725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.488599 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:48.492720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.489226 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:19:48.528559 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.526908 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:48.733730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.732156 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 20:19:48.762723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.762235 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.762723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.762292 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:48.794776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.784797 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.794776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.784856 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:48.808151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.808042 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.808151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.808096 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:48.883741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.877575 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:48.883741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.877638 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:48.958744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:48.955622 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 20:19:49.155450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.155410 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:19:49.167723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.156200 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:19:49.496963 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.496852 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:49.742725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.742117 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:49.742725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.742192 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:49.766805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.743570 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:49.779925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.779890 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:49.780194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.780162 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:49.820912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.820876 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:49.821167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.821142 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:49.822743 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.822718 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:49.823240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.823214 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:49.850431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.850394 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:19:49.851294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.851250 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" podUID="92fdeca4-c61c-4c03-9044-79186985b53c" containerName="maas-api" containerID="cri-o://0641a75ed04387478e968a09a1f20e73dcaba1dd15400893eb7e10b95243b878" gracePeriod=30 Oct 30 20:19:49.969229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:49.964260 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:50.035806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.035653 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:50.250720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.202932 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:50.545081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.545047 4524 generic.go:356] "Generic (PLEG): container finished" podID="92fdeca4-c61c-4c03-9044-79186985b53c" containerID="0641a75ed04387478e968a09a1f20e73dcaba1dd15400893eb7e10b95243b878" exitCode=0 Oct 30 20:19:50.545367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.545273 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" event={"ID":"92fdeca4-c61c-4c03-9044-79186985b53c","Type":"ContainerDied","Data":"0641a75ed04387478e968a09a1f20e73dcaba1dd15400893eb7e10b95243b878"} Oct 30 20:19:50.612253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.612168 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:19:50.697725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.697528 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:50.732886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.732846 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:50.733167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.733133 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:50.747831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.747785 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:50.827722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.814277 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:50.827722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.814336 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:50.861632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.860817 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:50.861632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.860879 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:50.861632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.860964 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:50.861632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.860988 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:50.966739 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:50.965984 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:51.122574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.122539 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:19:51.739932 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.732970 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:51.831117 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.831072 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:51.831407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.831381 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:51.855785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.831932 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:51.856867 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.856828 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:51.872627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.872591 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:51.872892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.872862 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:51.884193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.884157 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:51.884422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.884392 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:51.886330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.885481 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:51.944167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.944090 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:19:51.959318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:51.959266 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" podUID="b02608af-9cf6-4723-a99e-0fb13d333870" containerName="payload-processing" containerID="cri-o://e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081" gracePeriod=25 Oct 30 20:19:52.042144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.042102 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") pod \"92fdeca4-c61c-4c03-9044-79186985b53c\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " Oct 30 20:19:52.042537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.042511 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\") pod \"92fdeca4-c61c-4c03-9044-79186985b53c\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " Oct 30 20:19:52.042732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.042694 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\" (UniqueName: \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\") pod \"92fdeca4-c61c-4c03-9044-79186985b53c\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " Oct 30 20:19:52.042898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.042881 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\") pod \"92fdeca4-c61c-4c03-9044-79186985b53c\" (UID: \"92fdeca4-c61c-4c03-9044-79186985b53c\") " Oct 30 20:19:52.043629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.043593 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca" pod "92fdeca4-c61c-4c03-9044-79186985b53c" (UID: "92fdeca4-c61c-4c03-9044-79186985b53c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:52.069732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.069685 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fdeca4-c61c-4c03-9044-79186985b53c-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:52.076169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.076089 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs" pod "92fdeca4-c61c-4c03-9044-79186985b53c" (UID: "92fdeca4-c61c-4c03-9044-79186985b53c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:52.114298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.100177 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls" pod "92fdeca4-c61c-4c03-9044-79186985b53c" (UID: "92fdeca4-c61c-4c03-9044-79186985b53c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:52.133719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.129089 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:52.141132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.140218 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n" pod "92fdeca4-c61c-4c03-9044-79186985b53c" (UID: "92fdeca4-c61c-4c03-9044-79186985b53c"). InnerVolumeSpecName "kube-api-access-xs46n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:52.238613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.237595 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:19:52.340216 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.340176 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:52.340622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.340596 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs46n\" (UniqueName: \"kubernetes.io/projected/92fdeca4-c61c-4c03-9044-79186985b53c-kube-api-access-xs46n\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:52.342055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.340864 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fdeca4-c61c-4c03-9044-79186985b53c-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:52.347085 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.347058 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:19:52.600351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.599851 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:52.729151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.729114 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:52.729403 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.729378 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:52.781801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.779326 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:19:52.781801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.779974 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:52.781801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.780019 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:52.791732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.789238 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:52.791732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.789306 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:52.806666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.806642 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:52.806822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.806675 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:52.881771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.880970 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\" (UniqueName: \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\") pod \"b02608af-9cf6-4723-a99e-0fb13d333870\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " Oct 30 20:19:52.881771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.881052 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\") pod \"b02608af-9cf6-4723-a99e-0fb13d333870\" (UID: \"b02608af-9cf6-4723-a99e-0fb13d333870\") " Oct 30 20:19:52.882681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.882046 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume" pod "b02608af-9cf6-4723-a99e-0fb13d333870" (UID: "b02608af-9cf6-4723-a99e-0fb13d333870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:52.902736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.896842 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl" pod "b02608af-9cf6-4723-a99e-0fb13d333870" (UID: "b02608af-9cf6-4723-a99e-0fb13d333870"). InnerVolumeSpecName "kube-api-access-n7jtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:52.920880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.920844 4524 generic.go:356] "Generic (PLEG): container finished" podID="b02608af-9cf6-4723-a99e-0fb13d333870" containerID="e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081" exitCode=0 Oct 30 20:19:52.921049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.920917 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" Oct 30 20:19:52.921049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.920956 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" event={"ID":"b02608af-9cf6-4723-a99e-0fb13d333870","Type":"ContainerDied","Data":"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081"} Oct 30 20:19:52.921049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.921007 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8" event={"ID":"b02608af-9cf6-4723-a99e-0fb13d333870","Type":"ContainerDied","Data":"e68050dfd2158613c07c26ff70c84c8c1f96c2eea397100643f78e67d2925445"} Oct 30 20:19:52.921049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.921038 4524 scope.go:122] "RemoveContainer" containerID="e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081" Oct 30 20:19:52.925983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.924426 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" event={"ID":"92fdeca4-c61c-4c03-9044-79186985b53c","Type":"ContainerDied","Data":"c77aaee42cafe278d0031697703470d40d8084d76569b63677415b8d438bc458"} Oct 30 20:19:52.925983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.924536 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9" Oct 30 20:19:52.940973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.937561 4524 scope.go:122] "RemoveContainer" containerID="e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081" Oct 30 20:19:52.940973 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:52.938394 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081\": container with ID starting with e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081 not found: ID does not exist" containerID="e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081" Oct 30 20:19:52.940973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.938429 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081"} err="failed to get container status \"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081\": rpc error: code = NotFound desc = could not find container \"e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081\": container with ID starting with e12d19ddbada21569910f77aafaea09db404b88887463fa2ee3297c20a2e7081 not found: ID does not exist" Oct 30 20:19:52.940973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.938452 4524 scope.go:122] "RemoveContainer" containerID="0641a75ed04387478e968a09a1f20e73dcaba1dd15400893eb7e10b95243b878" Oct 30 20:19:53.009741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.992029 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jtl\" (UniqueName: \"kubernetes.io/projected/b02608af-9cf6-4723-a99e-0fb13d333870-kube-api-access-n7jtl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.010023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.009998 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02608af-9cf6-4723-a99e-0fb13d333870-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.010141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:52.998421 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.025066 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.010760 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" podUID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" containerName="payload-pre-processing" containerID="cri-o://7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff" gracePeriod=25 Oct 30 20:19:53.094220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.094174 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:19:53.100884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.100851 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25cc2f-5fc75bbd64-g7hc8"] Oct 30 20:19:53.208589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.208553 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:19:53.214979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.214953 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f44ec2-6c46cfb89c-6nwz9"] Oct 30 20:19:53.228682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.228651 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.308722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.308546 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fdeca4-c61c-4c03-9044-79186985b53c" path="/var/lib/kubelet/pods/92fdeca4-c61c-4c03-9044-79186985b53c/volumes" Oct 30 20:19:53.309606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.309129 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02608af-9cf6-4723-a99e-0fb13d333870" path="/var/lib/kubelet/pods/b02608af-9cf6-4723-a99e-0fb13d333870/volumes" Oct 30 20:19:53.381229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.380908 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:19:53.425027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.424891 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\" (UniqueName: \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\") pod \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " Oct 30 20:19:53.425027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.424951 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\") pod \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\" (UID: \"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2\") " Oct 30 20:19:53.426132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.426098 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume" pod "adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" (UID: "adebf0f7-b53d-42de-aa0a-41ce8a8b95b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:53.428394 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.428368 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw" pod "adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" (UID: "adebf0f7-b53d-42de-aa0a-41ce8a8b95b2"). InnerVolumeSpecName "kube-api-access-47lfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:53.528219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.526735 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lfw\" (UniqueName: \"kubernetes.io/projected/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-kube-api-access-47lfw\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.528219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.526767 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.632306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.632148 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.733053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.729523 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-q28fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 20:19:53.733053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.729578 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 20:19:53.780354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.780247 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 20:19:53.780354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.780302 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 20:19:53.788954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.788850 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 20:19:53.788954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.788886 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 20:19:53.794510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.794296 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:19:53.805846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.805782 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 20:19:53.805846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.805813 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 20:19:53.830440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830416 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830463 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830516 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830557 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830621 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830669 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830783 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830878 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.830980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.830938 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\") pod \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\" (UID: \"e3aa1748-6984-48c4-a060-6c0ee1d05c3b\") " Oct 30 20:19:53.831622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.831283 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:53.831622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.831523 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.831833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.831757 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:53.832244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.832210 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:53.832432 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.832397 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:53.832520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.832461 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:53.834501 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.834019 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "kube-api-access-zsvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:53.834912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.834888 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:53.835016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.834987 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:19:53.835234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.835211 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy" pod "e3aa1748-6984-48c4-a060-6c0ee1d05c3b" (UID: "e3aa1748-6984-48c4-a060-6c0ee1d05c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:53.943376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.943334 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.944918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944894 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.944918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944917 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.945097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944931 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.945097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944943 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.945097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944957 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.945097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944969 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.945097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.944983 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsvvn\" (UniqueName: \"kubernetes.io/projected/e3aa1748-6984-48c4-a060-6c0ee1d05c3b-kube-api-access-zsvvn\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:19:53.948952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.948924 4524 generic.go:356] "Generic (PLEG): container finished" podID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerID="04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0" exitCode=0 Oct 30 20:19:53.949389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.949175 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.949195 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerDied","Data":"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0"} Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.964066 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5" event={"ID":"e3aa1748-6984-48c4-a060-6c0ee1d05c3b","Type":"ContainerDied","Data":"775dc62d9793f23b62425806dee7d63243e7306ac56f18ddb93a94805a3500e6"} Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.964106 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" event={"ID":"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2","Type":"ContainerDied","Data":"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff"} Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.953025 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.952973 4524 generic.go:356] "Generic (PLEG): container finished" podID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" containerID="7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff" exitCode=0 Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.968067 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp" event={"ID":"adebf0f7-b53d-42de-aa0a-41ce8a8b95b2","Type":"ContainerDied","Data":"39236287cea6de9e2b8531f9f5c091ec57e1c73ddd829ae930dbadd682615201"} Oct 30 20:19:53.983679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:53.968117 4524 scope.go:122] "RemoveContainer" containerID="04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0" Oct 30 20:19:54.011195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.011170 4524 scope.go:122] "RemoveContainer" containerID="f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a" Oct 30 20:19:54.028767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.028686 4524 scope.go:122] "RemoveContainer" containerID="04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0" Oct 30 20:19:54.029185 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:54.029114 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0\": container with ID starting with 04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0 not found: ID does not exist" containerID="04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0" Oct 30 20:19:54.029430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.029401 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0"} err="failed to get container status \"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0\": rpc error: code = NotFound desc = could not find container \"04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0\": container with ID starting with 04aea85a2e48175e7e6995c851cf0dab42f4e833423b1bdb71f2de6dd14a68f0 not found: ID does not exist" Oct 30 20:19:54.029549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.029536 4524 scope.go:122] "RemoveContainer" containerID="f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a" Oct 30 20:19:54.030059 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:54.030000 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a\": container with ID starting with f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a not found: ID does not exist" containerID="f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a" Oct 30 20:19:54.030059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.030034 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a"} err="failed to get container status \"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a\": rpc error: code = NotFound desc = could not find container \"f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a\": container with ID starting with f0d714d51c0eb74d3779f098faa8f4f0762329f65a9c6a80ff9d3d152592501a not found: ID does not exist" Oct 30 20:19:54.030239 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.030072 4524 scope.go:122] "RemoveContainer" containerID="7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff" Oct 30 20:19:54.044354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.044331 4524 scope.go:122] "RemoveContainer" containerID="7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff" Oct 30 20:19:54.045217 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:19:54.045165 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff\": container with ID starting with 7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff not found: ID does not exist" containerID="7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff" Oct 30 20:19:54.045217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.045197 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff"} err="failed to get container status \"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff\": rpc error: code = NotFound desc = could not find container \"7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff\": container with ID starting with 7df7e2a5752b2f7d26db575d7e60d69fd37b544997621353dc3efc0fd5d65dff not found: ID does not exist" Oct 30 20:19:54.137617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.137580 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:19:54.144871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.144842 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25cc2f-6759849779-mvdfp"] Oct 30 20:19:54.268794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.264846 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:19:54.268794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:54.266247 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb565e-openshift-default-6b6f776d89-wrpt5"] Oct 30 20:19:55.477723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.470014 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" path="/var/lib/kubelet/pods/adebf0f7-b53d-42de-aa0a-41ce8a8b95b2/volumes" Oct 30 20:19:55.477723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.470922 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" path="/var/lib/kubelet/pods/e3aa1748-6984-48c4-a060-6c0ee1d05c3b/volumes" Oct 30 20:19:55.651990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.651961 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:55.654862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.654836 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:55.686049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.686009 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:19:55.823765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.822873 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-v95qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:19:55.823765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.822987 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:55.831942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.831866 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:19:55.831942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.831932 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:55.854719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.853778 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t624r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:19:55.854719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:55.853839 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:56.040518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.038971 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:56.040716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.040683 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:56.078231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.078141 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:19:56.250926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.250895 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:56.251620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.251596 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:56.252443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.252422 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:19:56.350238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.348608 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:56.350238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.349762 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:56.350927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:56.350906 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:19:58.593459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:58.593426 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:59.701629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:59.701576 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:59.929585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:19:59.929550 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:00.177628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:00.177579 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:01.033080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:01.033023 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:01.257413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:01.257379 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:03.920804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.920756 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:20:03.921413 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921395 4524 cpu_manager.go:414] "containerMap: removing container" podUID="92fdeca4-c61c-4c03-9044-79186985b53c" containerName="maas-api" Oct 30 20:20:03.921466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921415 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fdeca4-c61c-4c03-9044-79186985b53c" containerName="maas-api" Oct 30 20:20:03.921466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921435 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:03.921466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921440 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:03.921466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921450 4524 cpu_manager.go:414] "containerMap: removing container" podUID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" containerName="payload-pre-processing" Oct 30 20:20:03.921466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921456 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" containerName="payload-pre-processing" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921476 4524 cpu_manager.go:414] "containerMap: removing container" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921482 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921493 4524 cpu_manager.go:414] "containerMap: removing container" podUID="b02608af-9cf6-4723-a99e-0fb13d333870" containerName="payload-processing" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921498 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02608af-9cf6-4723-a99e-0fb13d333870" containerName="payload-processing" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921583 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fdeca4-c61c-4c03-9044-79186985b53c" containerName="maas-api" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921593 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921601 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="adebf0f7-b53d-42de-aa0a-41ce8a8b95b2" containerName="payload-pre-processing" Oct 30 20:20:03.921628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.921615 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02608af-9cf6-4723-a99e-0fb13d333870" containerName="payload-processing" Oct 30 20:20:03.926599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.926584 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:03.980993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.980968 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0051fb-serving-cert\"" Oct 30 20:20:03.986033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:03.986013 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:20:04.016301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.016266 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tl2d\" (UniqueName: \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.016525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.016506 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.016646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.016622 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.016817 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.016793 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.117934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.117904 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.118083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.117947 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tl2d\" (UniqueName: \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.118083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.118055 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.118171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.118097 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.118223 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:04.118201 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-serving-cert: secret "maas-api-e2e-inf-a-0051fb-serving-cert" not found Oct 30 20:20:04.118302 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:04.118289 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls podName:6d788ee1-cd0c-486c-9ee4-3bd725f1433b nodeName:}" failed. No retries permitted until 2026-10-30 20:20:04.618267304 +0000 UTC m=+2299.461938920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls") pod "maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" (UID: "6d788ee1-cd0c-486c-9ee4-3bd725f1433b") : secret "maas-api-e2e-inf-a-0051fb-serving-cert" not found Oct 30 20:20:04.118587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.118567 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.120471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.120450 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.193554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.190470 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.197618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.197588 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tl2d\" (UniqueName: \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.440780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.440731 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.627264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.627175 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.629627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.629603 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:04.840928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:04.840900 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:05.222847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:05.222819 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:20:05.224679 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:20:05.224645 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d788ee1_cd0c_486c_9ee4_3bd725f1433b.slice/crio-f3ce7361026a9858b43f6f26f7146d1c491bcc000ca974cd18a4336bc7eb7f7b WatchSource:0}: Error finding container f3ce7361026a9858b43f6f26f7146d1c491bcc000ca974cd18a4336bc7eb7f7b: Status 404 returned error can't find the container with id f3ce7361026a9858b43f6f26f7146d1c491bcc000ca974cd18a4336bc7eb7f7b Oct 30 20:20:05.226554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:05.226533 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:20:06.222720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:06.221727 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" event={"ID":"6d788ee1-cd0c-486c-9ee4-3bd725f1433b","Type":"ContainerStarted","Data":"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649"} Oct 30 20:20:06.222720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:06.221802 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" event={"ID":"6d788ee1-cd0c-486c-9ee4-3bd725f1433b","Type":"ContainerStarted","Data":"f3ce7361026a9858b43f6f26f7146d1c491bcc000ca974cd18a4336bc7eb7f7b"} Oct 30 20:20:06.239720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:06.236681 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:06.374048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:06.373944 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" podStartSLOduration=2.5199329539999997 podStartE2EDuration="3.373930837s" totalImagesPullingTime="853.997883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:06.372632852 +0000 UTC m=+2301.216304483" watchObservedRunningTime="2026-10-30 20:20:06.373930837 +0000 UTC m=+2301.217602466" Oct 30 20:20:08.895047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:08.895009 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:10.630451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:10.630410 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:11.053675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:11.053638 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:11.575729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:11.575677 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:11.634193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:11.634159 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:20:11.756093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:11.756061 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:20:11.822400 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:11.822359 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:12.091552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:12.091507 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:12.337465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:12.337430 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:12.790013 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:12.789974 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:20:13.261723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:13.261136 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:20:14.771675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:14.771638 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:14.982728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:14.982679 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:16.446008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:16.445974 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:16.635227 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:16.635184 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" podUID="0cea4c52-1a5c-4457-b387-b5dd65357a91" containerName="payload-processing" containerID="cri-o://186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208" gracePeriod=25 Oct 30 20:20:16.757204 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:16.757172 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" podUID="0bce1e32-198f-40d5-801d-542c4dae7282" containerName="payload-pre-processing" containerID="cri-o://5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3" gracePeriod=25 Oct 30 20:20:16.921825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:16.921797 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:20:17.039154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.039131 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:20:17.070504 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.070477 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\" (UniqueName: \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\") pod \"0cea4c52-1a5c-4457-b387-b5dd65357a91\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " Oct 30 20:20:17.070672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.070637 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\") pod \"0cea4c52-1a5c-4457-b387-b5dd65357a91\" (UID: \"0cea4c52-1a5c-4457-b387-b5dd65357a91\") " Oct 30 20:20:17.071110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.071074 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume" pod "0cea4c52-1a5c-4457-b387-b5dd65357a91" (UID: "0cea4c52-1a5c-4457-b387-b5dd65357a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:17.072653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.072624 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k" pod "0cea4c52-1a5c-4457-b387-b5dd65357a91" (UID: "0cea4c52-1a5c-4457-b387-b5dd65357a91"). InnerVolumeSpecName "kube-api-access-k5x9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:17.172174 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.172143 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\" (UniqueName: \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\") pod \"0bce1e32-198f-40d5-801d-542c4dae7282\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " Oct 30 20:20:17.172346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.172297 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\") pod \"0bce1e32-198f-40d5-801d-542c4dae7282\" (UID: \"0bce1e32-198f-40d5-801d-542c4dae7282\") " Oct 30 20:20:17.172930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.172902 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cea4c52-1a5c-4457-b387-b5dd65357a91-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:17.172930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.172930 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5x9k\" (UniqueName: \"kubernetes.io/projected/0cea4c52-1a5c-4457-b387-b5dd65357a91-kube-api-access-k5x9k\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:17.173323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.173297 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume" pod "0bce1e32-198f-40d5-801d-542c4dae7282" (UID: "0bce1e32-198f-40d5-801d-542c4dae7282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:17.175198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.175167 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz" pod "0bce1e32-198f-40d5-801d-542c4dae7282" (UID: "0bce1e32-198f-40d5-801d-542c4dae7282"). InnerVolumeSpecName "kube-api-access-jxmhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:17.274386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.274310 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bce1e32-198f-40d5-801d-542c4dae7282-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:17.274386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.274341 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmhz\" (UniqueName: \"kubernetes.io/projected/0bce1e32-198f-40d5-801d-542c4dae7282-kube-api-access-jxmhz\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:17.442451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.442417 4524 generic.go:356] "Generic (PLEG): container finished" podID="0cea4c52-1a5c-4457-b387-b5dd65357a91" containerID="186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208" exitCode=0 Oct 30 20:20:17.442627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.442466 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" Oct 30 20:20:17.442627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.442541 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" event={"ID":"0cea4c52-1a5c-4457-b387-b5dd65357a91","Type":"ContainerDied","Data":"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208"} Oct 30 20:20:17.442627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.442594 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff" event={"ID":"0cea4c52-1a5c-4457-b387-b5dd65357a91","Type":"ContainerDied","Data":"ba5807c4a30716d28310e6c1efeae72d236538a6083a2d368c827994c2e4f682"} Oct 30 20:20:17.442627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.442620 4524 scope.go:122] "RemoveContainer" containerID="186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208" Oct 30 20:20:17.443930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.443903 4524 generic.go:356] "Generic (PLEG): container finished" podID="0bce1e32-198f-40d5-801d-542c4dae7282" containerID="5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3" exitCode=0 Oct 30 20:20:17.444048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.444029 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" event={"ID":"0bce1e32-198f-40d5-801d-542c4dae7282","Type":"ContainerDied","Data":"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3"} Oct 30 20:20:17.444094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.444078 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" event={"ID":"0bce1e32-198f-40d5-801d-542c4dae7282","Type":"ContainerDied","Data":"b3d3b6a71a9bfad9c9a90d997f77275491372c8e05dab720c2b051abee39afde"} Oct 30 20:20:17.444165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.444152 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx" Oct 30 20:20:17.455226 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.455207 4524 scope.go:122] "RemoveContainer" containerID="186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208" Oct 30 20:20:17.455519 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:17.455492 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208\": container with ID starting with 186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208 not found: ID does not exist" containerID="186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208" Oct 30 20:20:17.455561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.455528 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208"} err="failed to get container status \"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208\": rpc error: code = NotFound desc = could not find container \"186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208\": container with ID starting with 186a7db54c57ab5441d98d7dcb01ff34203019b4a0e811a9b717fc637ac90208 not found: ID does not exist" Oct 30 20:20:17.455561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.455550 4524 scope.go:122] "RemoveContainer" containerID="5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3" Oct 30 20:20:17.466479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.466455 4524 scope.go:122] "RemoveContainer" containerID="5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3" Oct 30 20:20:17.466820 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:17.466794 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3\": container with ID starting with 5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3 not found: ID does not exist" containerID="5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3" Oct 30 20:20:17.466912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.466828 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3"} err="failed to get container status \"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3\": rpc error: code = NotFound desc = could not find container \"5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3\": container with ID starting with 5bd87951e80dfc4454331d74bc0b0a0a437461cf0cd064f9fc37b029812704b3 not found: ID does not exist" Oct 30 20:20:17.611031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.610964 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:20:17.660182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.660154 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dc56e1-7c7b6db8c-9mrmx"] Oct 30 20:20:17.724975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.724946 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:20:17.730168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:17.730144 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dc56e1-67588d96c4-cgbff"] Oct 30 20:20:18.693079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:18.692970 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:18.917345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:18.917313 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:18.996559 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:18.996483 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:20:18.996772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:18.996747 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" containerID="cri-o://67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d" gracePeriod=30 Oct 30 20:20:19.137332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:19.137274 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:19.300855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:19.300777 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bce1e32-198f-40d5-801d-542c4dae7282" path="/var/lib/kubelet/pods/0bce1e32-198f-40d5-801d-542c4dae7282/volumes" Oct 30 20:20:19.301245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:19.301231 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cea4c52-1a5c-4457-b387-b5dd65357a91" path="/var/lib/kubelet/pods/0cea4c52-1a5c-4457-b387-b5dd65357a91/volumes" Oct 30 20:20:20.898289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:20.898249 4524 patch_prober.go:29] interesting pod/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:20:20.898761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:20.898326 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:20:21.529726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:21.529056 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:24.263362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.263304 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:20:24.385301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385267 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385337 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385407 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385459 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385581 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385621 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:24.385760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385678 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385746 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385784 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.385888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385873 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\") pod \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\" (UID: \"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347\") " Oct 30 20:20:24.386000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385943 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:24.386000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.385872 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:24.386201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386141 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:24.386376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386352 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:24.386542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386520 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.386683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386547 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.386683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386564 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.386683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386578 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.386683 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.386592 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.387801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.387779 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:24.388035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.388009 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "kube-api-access-zw7dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:24.388280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.388257 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:20:24.388394 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.388271 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token" pod "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" (UID: "1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:24.487847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.487815 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.488075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.488046 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.488075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.488071 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7dq\" (UniqueName: \"kubernetes.io/projected/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-kube-api-access-zw7dq\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.488222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.488087 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:24.490531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.490502 4524 generic.go:356] "Generic (PLEG): container finished" podID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerID="67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d" exitCode=0 Oct 30 20:20:24.490634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.490555 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" Oct 30 20:20:24.490634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.490584 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerDied","Data":"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d"} Oct 30 20:20:24.490769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.490637 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb" event={"ID":"1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347","Type":"ContainerDied","Data":"f04bf55ffc9248c065d0d5101f61e537aeef2fa8205b7c359e42d9392302c67f"} Oct 30 20:20:24.490769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.490665 4524 scope.go:122] "RemoveContainer" containerID="67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d" Oct 30 20:20:24.500810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.500790 4524 scope.go:122] "RemoveContainer" containerID="fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09" Oct 30 20:20:24.509291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.509275 4524 scope.go:122] "RemoveContainer" containerID="67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d" Oct 30 20:20:24.509524 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:24.509505 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d\": container with ID starting with 67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d not found: ID does not exist" containerID="67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d" Oct 30 20:20:24.509596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.509536 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d"} err="failed to get container status \"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d\": rpc error: code = NotFound desc = could not find container \"67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d\": container with ID starting with 67e09a784747550bd8aeb9c388542a37338d22e7d9d9068f1dd12dec7b9bee2d not found: ID does not exist" Oct 30 20:20:24.509596 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.509559 4524 scope.go:122] "RemoveContainer" containerID="fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09" Oct 30 20:20:24.510385 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:24.509804 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09\": container with ID starting with fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09 not found: ID does not exist" containerID="fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09" Oct 30 20:20:24.510385 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.509833 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09"} err="failed to get container status \"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09\": rpc error: code = NotFound desc = could not find container \"fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09\": container with ID starting with fdc950d6c22d217491fcf30f0689d61b3631d378d5a826ea3b5641492f4bdb09 not found: ID does not exist" Oct 30 20:20:24.656454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.656425 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:20:24.663110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.663087 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-770dfe-openshift-default-7dbfd8f78f-zc8qb"] Oct 30 20:20:24.751021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.750987 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:20:24.751273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:24.751229 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" containerID="cri-o://ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b" gracePeriod=30 Oct 30 20:20:25.300562 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:25.300530 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" path="/var/lib/kubelet/pods/1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347/volumes" Oct 30 20:20:27.030723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:27.027315 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:29.940442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:29.940109 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:30.157966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.157940 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:20:30.197130 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197105 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197145 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197213 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197321 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197350 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197397 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197437 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197557 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197589 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\") pod \"74a5eb43-b935-4ffe-8044-388cb9267850\" (UID: \"74a5eb43-b935-4ffe-8044-388cb9267850\") " Oct 30 20:20:30.197690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.197599 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:30.199502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.198241 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.199502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.199221 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:30.199502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.199417 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:30.199502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.199476 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:30.205565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.204791 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:30.205759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.205738 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:30.205853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.205836 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:30.207074 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.207047 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:20:30.208495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.208466 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z" pod "74a5eb43-b935-4ffe-8044-388cb9267850" (UID: "74a5eb43-b935-4ffe-8044-388cb9267850"). InnerVolumeSpecName "kube-api-access-jr95z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:30.224909 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.224885 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:30.299230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299204 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299230 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a5eb43-b935-4ffe-8044-388cb9267850-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299239 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299248 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299257 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a5eb43-b935-4ffe-8044-388cb9267850-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299265 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299276 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a5eb43-b935-4ffe-8044-388cb9267850-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.299459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.299286 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr95z\" (UniqueName: \"kubernetes.io/projected/74a5eb43-b935-4ffe-8044-388cb9267850-kube-api-access-jr95z\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:30.529551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.529516 4524 generic.go:356] "Generic (PLEG): container finished" podID="74a5eb43-b935-4ffe-8044-388cb9267850" containerID="ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b" exitCode=0 Oct 30 20:20:30.529755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.529562 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" Oct 30 20:20:30.529755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.529596 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" event={"ID":"74a5eb43-b935-4ffe-8044-388cb9267850","Type":"ContainerDied","Data":"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b"} Oct 30 20:20:30.529755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.529651 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q" event={"ID":"74a5eb43-b935-4ffe-8044-388cb9267850","Type":"ContainerDied","Data":"85bcb6f5bb259a96cd86fdeea6cf769448981dad8c5aa02ba6c7620a5fb33976"} Oct 30 20:20:30.529755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.529680 4524 scope.go:122] "RemoveContainer" containerID="ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b" Oct 30 20:20:30.540550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.540532 4524 scope.go:122] "RemoveContainer" containerID="ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b" Oct 30 20:20:30.540872 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:30.540844 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b\": container with ID starting with ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b not found: ID does not exist" containerID="ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b" Oct 30 20:20:30.540964 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.540884 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b"} err="failed to get container status \"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b\": rpc error: code = NotFound desc = could not find container \"ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b\": container with ID starting with ca8a696402181c2712255f850efc188c2d39d8913f2bdb7ffb4b398846f5f67b not found: ID does not exist" Oct 30 20:20:30.696394 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.696364 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:20:30.746248 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:30.746211 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-4m64q"] Oct 30 20:20:31.301446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:31.301414 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" path="/var/lib/kubelet/pods/74a5eb43-b935-4ffe-8044-388cb9267850/volumes" Oct 30 20:20:31.889445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:31.889397 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.077509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.077466 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:20:32.078483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078456 4524 cpu_manager.go:414] "containerMap: removing container" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.078483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078480 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078504 4524 cpu_manager.go:414] "containerMap: removing container" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078510 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078524 4524 cpu_manager.go:414] "containerMap: removing container" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078532 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078550 4524 cpu_manager.go:414] "containerMap: removing container" podUID="0cea4c52-1a5c-4457-b387-b5dd65357a91" containerName="payload-processing" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078572 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cea4c52-1a5c-4457-b387-b5dd65357a91" containerName="payload-processing" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078608 4524 cpu_manager.go:414] "containerMap: removing container" podUID="0bce1e32-198f-40d5-801d-542c4dae7282" containerName="payload-pre-processing" Oct 30 20:20:32.078710 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078614 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bce1e32-198f-40d5-801d-542c4dae7282" containerName="payload-pre-processing" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078767 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bce1e32-198f-40d5-801d-542c4dae7282" containerName="payload-pre-processing" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078782 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aa1748-6984-48c4-a060-6c0ee1d05c3b" containerName="istio-proxy" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078799 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078810 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a5eb43-b935-4ffe-8044-388cb9267850" containerName="istio-proxy" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078818 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5f34ff-8ed9-4a77-a66a-2ce8f71b9347" containerName="istio-proxy" Oct 30 20:20:32.079138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.078829 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cea4c52-1a5c-4457-b387-b5dd65357a91" containerName="payload-processing" Oct 30 20:20:32.083951 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.083930 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.111858 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.111825 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.121841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.121814 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.121967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.121870 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.122020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.121972 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47zp\" (UniqueName: \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.122086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.122070 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.137372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.137340 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-37732de7-serving-cert\"" Oct 30 20:20:32.190233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.190196 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:20:32.223281 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.223240 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.223492 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.223458 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.223587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.223510 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.223587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.223559 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47zp\" (UniqueName: \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.224268 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:32.224248 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-serving-cert: secret "maas-api-e2e-mt-37732de7-serving-cert" not found Oct 30 20:20:32.224387 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:32.224323 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls podName:02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1 nodeName:}" failed. No retries permitted until 2026-10-30 20:20:32.724302117 +0000 UTC m=+2327.567973738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls") pod "maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" (UID: "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1") : secret "maas-api-e2e-mt-37732de7-serving-cert" not found Oct 30 20:20:32.225125 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.225094 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.227616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.227589 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.285495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.285462 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47zp\" (UniqueName: \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.405577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.405536 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.625447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.625328 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.739798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.739767 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.742184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.742151 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") pod \"maas-api-e2e-mt-37732de7-7b44f7679-cgbr2\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:32.999060 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:32.998982 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:33.450640 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:33.450609 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:20:33.452178 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:20:33.452142 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cd6e01_20ed_46eb_8bc6_cccf6bd3ebf1.slice/crio-80971d7ddff14937b53d5c25a245eeb56938df75028e9c9e9927324843b6e383 WatchSource:0}: Error finding container 80971d7ddff14937b53d5c25a245eeb56938df75028e9c9e9927324843b6e383: Status 404 returned error can't find the container with id 80971d7ddff14937b53d5c25a245eeb56938df75028e9c9e9927324843b6e383 Oct 30 20:20:33.553762 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:33.553726 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" event={"ID":"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1","Type":"ContainerStarted","Data":"80971d7ddff14937b53d5c25a245eeb56938df75028e9c9e9927324843b6e383"} Oct 30 20:20:34.563949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:34.563905 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" event={"ID":"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1","Type":"ContainerStarted","Data":"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b"} Oct 30 20:20:34.565027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:34.565004 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:34.684765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:34.684691 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" podStartSLOduration=1.8218135370000001 podStartE2EDuration="2.684676079s" totalImagesPullingTime="862.862542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:34.683291126 +0000 UTC m=+2329.526962768" watchObservedRunningTime="2026-10-30 20:20:34.684676079 +0000 UTC m=+2329.528347752" Oct 30 20:20:37.022625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.022577 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:20:37.023154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.022885 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" podUID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" containerName="maas-api" containerID="cri-o://450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a" gracePeriod=30 Oct 30 20:20:37.303880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.303787 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:37.355418 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.355387 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:20:37.450232 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.450182 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\") pod \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " Oct 30 20:20:37.450424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.450403 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\" (UniqueName: \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\") pod \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " Oct 30 20:20:37.450517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.450462 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") pod \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " Oct 30 20:20:37.450575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.450522 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\") pod \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\" (UID: \"59b7c55e-6ee8-433c-aab5-1c808334c9b8\") " Oct 30 20:20:37.464266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.464229 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca" pod "59b7c55e-6ee8-433c-aab5-1c808334c9b8" (UID: "59b7c55e-6ee8-433c-aab5-1c808334c9b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:37.483764 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.482431 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs" pod "59b7c55e-6ee8-433c-aab5-1c808334c9b8" (UID: "59b7c55e-6ee8-433c-aab5-1c808334c9b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:37.507721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.502823 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls" pod "59b7c55e-6ee8-433c-aab5-1c808334c9b8" (UID: "59b7c55e-6ee8-433c-aab5-1c808334c9b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:37.519946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.519905 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4" pod "59b7c55e-6ee8-433c-aab5-1c808334c9b8" (UID: "59b7c55e-6ee8-433c-aab5-1c808334c9b8"). InnerVolumeSpecName "kube-api-access-lr2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:37.539205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.539177 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:37.552973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.552947 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b7c55e-6ee8-433c-aab5-1c808334c9b8-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:37.553068 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.552976 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2l4\" (UniqueName: \"kubernetes.io/projected/59b7c55e-6ee8-433c-aab5-1c808334c9b8-kube-api-access-lr2l4\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:37.553068 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.552994 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:37.553068 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.553008 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b7c55e-6ee8-433c-aab5-1c808334c9b8-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:37.592760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.588784 4524 generic.go:356] "Generic (PLEG): container finished" podID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" containerID="450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a" exitCode=0 Oct 30 20:20:37.592760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.588837 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" Oct 30 20:20:37.592760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.588891 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" event={"ID":"59b7c55e-6ee8-433c-aab5-1c808334c9b8","Type":"ContainerDied","Data":"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a"} Oct 30 20:20:37.592760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.588935 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g" event={"ID":"59b7c55e-6ee8-433c-aab5-1c808334c9b8","Type":"ContainerDied","Data":"5e315531d2a46392b24c492026c832408e40f0d085c95fb31563c4a7f9841995"} Oct 30 20:20:37.592760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.588962 4524 scope.go:122] "RemoveContainer" containerID="450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a" Oct 30 20:20:37.654065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.653981 4524 scope.go:122] "RemoveContainer" containerID="450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a" Oct 30 20:20:37.654457 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:37.654426 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a\": container with ID starting with 450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a not found: ID does not exist" containerID="450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a" Oct 30 20:20:37.654569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.654472 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a"} err="failed to get container status \"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a\": rpc error: code = NotFound desc = could not find container \"450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a\": container with ID starting with 450dd6b609f2ea711f132a039219493e8a1b1f0b4fbfa60502533cacd6de090a not found: ID does not exist" Oct 30 20:20:37.774591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.774560 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:20:37.779228 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:37.779198 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25cc2f-676d597b4c-g5t9g"] Oct 30 20:20:38.819947 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:38.819913 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.118812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:39.115987 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.301357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:39.301323 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" path="/var/lib/kubelet/pods/59b7c55e-6ee8-433c-aab5-1c808334c9b8/volumes" Oct 30 20:20:39.491393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:39.491357 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.910381 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:39.910343 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:41.599278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:41.599249 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:20:42.797488 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:42.797448 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:20:44.317061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:44.317020 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:44.606822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:44.605514 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:46.868725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:46.867721 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.125141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:47.125047 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.237411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:47.237381 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:20:47.237602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:47.237579 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" containerID="cri-o://11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5" gracePeriod=30 Oct 30 20:20:47.388784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:47.383511 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.721805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:47.721769 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:48.303631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:48.303548 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:49.619550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:49.619510 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:49.870662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:49.870565 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:50.143089 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.143052 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:20:50.143722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.143676 4524 cpu_manager.go:414] "containerMap: removing container" podUID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" containerName="maas-api" Oct 30 20:20:50.143722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.143712 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" containerName="maas-api" Oct 30 20:20:50.143902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.143887 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b7c55e-6ee8-433c-aab5-1c808334c9b8" containerName="maas-api" Oct 30 20:20:50.149255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.149235 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.203914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.203877 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-01e8dc-openshift-default-dockercfg-m7b5x\"" Oct 30 20:20:50.230233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.230181 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:20:50.247337 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247300 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247514 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247357 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247521 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247585 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247626 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqc5\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247770 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.247933 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247913 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.248020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.247955 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.248064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.248042 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.318020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.317913 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.351959 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352032 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352066 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352106 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352215 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352256 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352337 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352378 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.353234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.352408 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqc5\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.373719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.365501 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.373719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.365891 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.373719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.366162 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.373719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.366352 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.387723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.376998 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.448801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.448312 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.454637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.453514 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.486896 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.485750 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.505512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.504424 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqc5\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\") pod \"e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:50.537187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.537080 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:50.770918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:50.770845 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:51.212225 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.212193 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:20:51.212352 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:20:51.212254 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5238180_2ad7_4b20_bec7_5c244c68c2d0.slice/crio-c71df95b473605468d4f0d2bb847620937d76b2a5148b4b186c25e3265d1e339 WatchSource:0}: Error finding container c71df95b473605468d4f0d2bb847620937d76b2a5148b4b186c25e3265d1e339: Status 404 returned error can't find the container with id c71df95b473605468d4f0d2bb847620937d76b2a5148b4b186c25e3265d1e339 Oct 30 20:20:51.215037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.215007 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:51.215136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.215068 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:51.215136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.215100 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:51.708607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.708479 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerStarted","Data":"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554"} Oct 30 20:20:51.708607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.708535 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerStarted","Data":"c71df95b473605468d4f0d2bb847620937d76b2a5148b4b186c25e3265d1e339"} Oct 30 20:20:51.770904 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:51.770883 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:52.522374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.522348 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:20:52.582380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582346 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582482 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582518 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582584 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:52.582659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582637 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582667 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582747 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582830 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582811 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.582939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582909 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.583019 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.582933 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:52.583097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583074 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\") pod \"844d0230-0706-4c0e-a80c-1133fdc14d35\" (UID: \"844d0230-0706-4c0e-a80c-1133fdc14d35\") " Oct 30 20:20:52.583164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583082 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:52.583269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583243 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:52.583479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583442 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:52.583822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583797 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.583927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583826 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.583927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583842 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.583927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583856 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844d0230-0706-4c0e-a80c-1133fdc14d35-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.583927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.583869 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.585021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.584996 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:52.585021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.585010 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:20:52.585305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.585291 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "kube-api-access-95lxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:52.585399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.585385 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token" pod "844d0230-0706-4c0e-a80c-1133fdc14d35" (UID: "844d0230-0706-4c0e-a80c-1133fdc14d35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:52.641058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.641034 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:52.641237 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.641078 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:52.685302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.685273 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lxp\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-kube-api-access-95lxp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.685302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.685301 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.685453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.685310 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.685453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.685320 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844d0230-0706-4c0e-a80c-1133fdc14d35-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:52.717173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.717145 4524 generic.go:356] "Generic (PLEG): container finished" podID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerID="5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5" exitCode=0 Oct 30 20:20:52.717332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.717221 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerDied","Data":"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5"} Oct 30 20:20:52.717663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.717646 4524 scope.go:122] "RemoveContainer" containerID="5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5" Oct 30 20:20:52.719191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719165 4524 generic.go:356] "Generic (PLEG): container finished" podID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerID="11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5" exitCode=0 Oct 30 20:20:52.719301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719209 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" Oct 30 20:20:52.719301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719250 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerDied","Data":"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5"} Oct 30 20:20:52.719301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719280 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:52.719442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719303 4524 scope.go:122] "RemoveContainer" containerID="11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5" Oct 30 20:20:52.719442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719345 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:52.719442 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719375 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:52.719608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.719294 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr" event={"ID":"844d0230-0706-4c0e-a80c-1133fdc14d35","Type":"ContainerDied","Data":"abb6a3fc23560ba54c815470b3d31b33e3246e61d13afcf172dc2f14fe4c9ba6"} Oct 30 20:20:52.732022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.732001 4524 scope.go:122] "RemoveContainer" containerID="f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7" Oct 30 20:20:52.742629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.742606 4524 scope.go:122] "RemoveContainer" containerID="11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5" Oct 30 20:20:52.742974 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:52.742940 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5\": container with ID starting with 11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5 not found: ID does not exist" containerID="11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5" Oct 30 20:20:52.743081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.742982 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5"} err="failed to get container status \"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5\": rpc error: code = NotFound desc = could not find container \"11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5\": container with ID starting with 11e2fee3d74ff389342bbe71a798b5d20fcc56bd666bc3ec5923bd36be281ad5 not found: ID does not exist" Oct 30 20:20:52.743081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.743001 4524 scope.go:122] "RemoveContainer" containerID="f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7" Oct 30 20:20:52.743294 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:52.743269 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7\": container with ID starting with f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7 not found: ID does not exist" containerID="f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7" Oct 30 20:20:52.743393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.743303 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7"} err="failed to get container status \"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7\": rpc error: code = NotFound desc = could not find container \"f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7\": container with ID starting with f91a38a9d0805a19950b83c0976dff1b6901a47d60301593ca5bd7489d42c5a7 not found: ID does not exist" Oct 30 20:20:52.746129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.746077 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podStartSLOduration=2.746060024 podStartE2EDuration="2.746060024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:51.938620958 +0000 UTC m=+2346.782292584" watchObservedRunningTime="2026-10-30 20:20:52.746060024 +0000 UTC m=+2347.589731655" Oct 30 20:20:52.778261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:52.778198 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:53.006735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.006678 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:20:53.023808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.023744 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-25cc2f-openshift-default-67fd57c899-z7mlr"] Oct 30 20:20:53.301878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.301802 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" path="/var/lib/kubelet/pods/844d0230-0706-4c0e-a80c-1133fdc14d35/volumes" Oct 30 20:20:53.728466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.728282 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerStarted","Data":"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9"} Oct 30 20:20:53.729924 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.729900 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:53.730808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:53.730790 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:20:54.631691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:54.631579 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:54.685381 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:54.685357 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:20:54.685607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:54.685532 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:54.685744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:54.685726 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:54.868724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:54.865048 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.104577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:55.104539 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.685296 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:55.685258 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:55.685474 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:55.685321 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:56.685107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:56.685069 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:56.685588 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:56.685134 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:57.685822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:57.685786 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:57.686318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:57.685858 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:58.237895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:58.237856 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:58.473741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:58.465571 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:58.686254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:58.685470 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 20:20:58.686254 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:58.685550 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 20:20:59.370911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.370880 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:20:59.371096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.371076 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" podUID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" containerName="maas-api" containerID="cri-o://e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08" gracePeriod=30 Oct 30 20:20:59.561093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.560799 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:59.692246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.691482 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:20:59.692246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.692205 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:20:59.693106 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.693081 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:20:59.712747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.712725 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:20:59.737679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.737652 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\") pod \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " Oct 30 20:20:59.737859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.737728 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\") pod \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " Oct 30 20:20:59.737859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.737779 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\" (UniqueName: \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\") pod \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " Oct 30 20:20:59.737859 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.737822 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") pod \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\" (UID: \"ac7bd72a-66dc-489c-b646-ea06c987b9f8\") " Oct 30 20:20:59.738103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.738078 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca" pod "ac7bd72a-66dc-489c-b646-ea06c987b9f8" (UID: "ac7bd72a-66dc-489c-b646-ea06c987b9f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:59.738429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.738393 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7bd72a-66dc-489c-b646-ea06c987b9f8-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:59.740051 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.740024 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj" pod "ac7bd72a-66dc-489c-b646-ea06c987b9f8" (UID: "ac7bd72a-66dc-489c-b646-ea06c987b9f8"). InnerVolumeSpecName "kube-api-access-z4rpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:59.740467 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.740448 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs" pod "ac7bd72a-66dc-489c-b646-ea06c987b9f8" (UID: "ac7bd72a-66dc-489c-b646-ea06c987b9f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:59.740515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.740467 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls" pod "ac7bd72a-66dc-489c-b646-ea06c987b9f8" (UID: "ac7bd72a-66dc-489c-b646-ea06c987b9f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:59.824823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.824788 4524 generic.go:356] "Generic (PLEG): container finished" podID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" containerID="e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08" exitCode=0 Oct 30 20:20:59.824965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.824834 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" Oct 30 20:20:59.824965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.824871 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" event={"ID":"ac7bd72a-66dc-489c-b646-ea06c987b9f8","Type":"ContainerDied","Data":"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08"} Oct 30 20:20:59.824965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.824921 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc" event={"ID":"ac7bd72a-66dc-489c-b646-ea06c987b9f8","Type":"ContainerDied","Data":"24a60a7a324ec7d1694933efe467ea7d718d6507ed54509198aed10adf51cf7b"} Oct 30 20:20:59.824965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.824943 4524 scope.go:122] "RemoveContainer" containerID="e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08" Oct 30 20:20:59.842765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.842725 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:59.842765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.842762 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7bd72a-66dc-489c-b646-ea06c987b9f8-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:59.843044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.842780 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rpj\" (UniqueName: \"kubernetes.io/projected/ac7bd72a-66dc-489c-b646-ea06c987b9f8-kube-api-access-z4rpj\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:20:59.918167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.918125 4524 scope.go:122] "RemoveContainer" containerID="e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08" Oct 30 20:20:59.918991 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:20:59.918965 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08\": container with ID starting with e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08 not found: ID does not exist" containerID="e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08" Oct 30 20:20:59.919194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:20:59.919168 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08"} err="failed to get container status \"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08\": rpc error: code = NotFound desc = could not find container \"e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08\": container with ID starting with e6ff78d1bb9866917eb12f358e566309a9ea48b6e5e197ce3a14eeb0d16b2e08 not found: ID does not exist" Oct 30 20:21:00.200759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:00.189089 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:21:00.238793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:00.237052 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dc56e1-8676b656d8-4lppc"] Oct 30 20:21:01.300470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:01.300433 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" path="/var/lib/kubelet/pods/ac7bd72a-66dc-489c-b646-ea06c987b9f8/volumes" Oct 30 20:21:01.303255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:01.303233 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:01.530014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:01.529980 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:01.825126 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:01.825087 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:02.363070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:02.363037 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.126721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:07.126431 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:10.069787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:10.069750 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:10.313288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:10.313250 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:10.563482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:10.563435 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:21:10.632724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:10.625033 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" containerID="cri-o://469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d" gracePeriod=30 Oct 30 20:21:10.922733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:10.919633 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.506726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:11.506668 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.770025 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:11.769941 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:12.008897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:12.008860 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:12.372255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:12.372219 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:12.825605 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:12.825568 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:21:13.281720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:13.281174 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.600391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:13.600290 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.826262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:13.826229 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:14.014311 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:14.014270 4524 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:21:14.014494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:14.014343 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:21:16.068115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.068084 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:21:16.177371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.177334 4524 generic.go:356] "Generic (PLEG): container finished" podID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerID="469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d" exitCode=0 Oct 30 20:21:16.177541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.177404 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" Oct 30 20:21:16.177541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.177420 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerDied","Data":"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d"} Oct 30 20:21:16.177541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.177517 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q" event={"ID":"fe3f3e93-8820-4edc-a483-fff28bb89580","Type":"ContainerDied","Data":"bbb4f8329107380d3d036454df9ac9ba8e9da4bd3773116756679375d0b0a2aa"} Oct 30 20:21:16.177677 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.177546 4524 scope.go:122] "RemoveContainer" containerID="469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d" Oct 30 20:21:16.189795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.189047 4524 scope.go:122] "RemoveContainer" containerID="4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200060 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200111 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200133 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200212 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200237 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200281 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200328 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200380 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.200427 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\") pod \"fe3f3e93-8820-4edc-a483-fff28bb89580\" (UID: \"fe3f3e93-8820-4edc-a483-fff28bb89580\") " Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.201144 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.201300 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.201437 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.203206 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.203415 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:16.203735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.203675 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:16.204676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.204191 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:21:16.205997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.205867 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:16.206122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.206088 4524 scope.go:122] "RemoveContainer" containerID="469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d" Oct 30 20:21:16.206412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.206385 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94" pod "fe3f3e93-8820-4edc-a483-fff28bb89580" (UID: "fe3f3e93-8820-4edc-a483-fff28bb89580"). InnerVolumeSpecName "kube-api-access-84z94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:16.206507 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:16.206446 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d\": container with ID starting with 469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d not found: ID does not exist" containerID="469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d" Oct 30 20:21:16.206507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.206481 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d"} err="failed to get container status \"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d\": rpc error: code = NotFound desc = could not find container \"469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d\": container with ID starting with 469e7fca5c13d93c05afdadabca9b64f73aeea8731a89948f3b4df3d1077915d not found: ID does not exist" Oct 30 20:21:16.206629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.206505 4524 scope.go:122] "RemoveContainer" containerID="4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9" Oct 30 20:21:16.206814 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:16.206796 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9\": container with ID starting with 4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9 not found: ID does not exist" containerID="4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9" Oct 30 20:21:16.206888 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.206822 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9"} err="failed to get container status \"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9\": rpc error: code = NotFound desc = could not find container \"4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9\": container with ID starting with 4abaaa71989a2c8d56cf46c3278f4a2d81d145a95feb378b3f7456b10a2d8bd9 not found: ID does not exist" Oct 30 20:21:16.301979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.301949 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.301979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.301975 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.301979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.301985 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.301994 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3f3e93-8820-4edc-a483-fff28bb89580-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.302003 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84z94\" (UniqueName: \"kubernetes.io/projected/fe3f3e93-8820-4edc-a483-fff28bb89580-kube-api-access-84z94\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.302012 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.302022 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.302030 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.302206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.302038 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3f3e93-8820-4edc-a483-fff28bb89580-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:16.601658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.601631 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:21:16.605566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:16.605541 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dc56e1-openshift-default-85c7bb79cc-fx88q"] Oct 30 20:21:17.104604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:17.104570 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:17.302332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:17.302297 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" path="/var/lib/kubelet/pods/fe3f3e93-8820-4edc-a483-fff28bb89580/volumes" Oct 30 20:21:17.547995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:17.547955 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:18.014288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:18.014245 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:19.863574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.863539 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:21:19.864351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864329 4524 cpu_manager.go:414] "containerMap: removing container" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:19.864409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864354 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:19.864409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864382 4524 cpu_manager.go:414] "containerMap: removing container" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:19.864409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864390 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:19.864409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864400 4524 cpu_manager.go:414] "containerMap: removing container" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:19.864409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864408 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:19.864568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864458 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" containerName="maas-api" Oct 30 20:21:19.864568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864465 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" containerName="maas-api" Oct 30 20:21:19.864629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864587 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:19.864629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864599 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:19.864629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.864614 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7bd72a-66dc-489c-b646-ea06c987b9f8" containerName="maas-api" Oct 30 20:21:19.869916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.869890 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.961597 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.961776 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.961828 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.961896 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.961921 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.962002 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbhr\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.962107 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.962167 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.962804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.962355 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:19.963140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.963114 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:19.986755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:19.986093 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.087736 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.087808 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.087879 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.087908 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.088006 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbhr\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.088118 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.089406 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.089727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.089688 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.090340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.090116 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.092657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.091484 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.093977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.093870 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.093977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.093940 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.094196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.094020 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.094439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.094413 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.097332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.097297 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.098997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.098974 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.152599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.151435 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbhr\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.152599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.152391 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.187053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.187026 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:20.188990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.188964 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:20.408326 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.408248 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:20.629753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.629683 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:21:20.641470 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:21:20.641438 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60983cc5_207e_4829_9473_2d91764280c1.slice/crio-85f223d37af74ec3d26239a3e78c684cb710c49516fb1343fb65971c8b0536a9 WatchSource:0}: Error finding container 85f223d37af74ec3d26239a3e78c684cb710c49516fb1343fb65971c8b0536a9: Status 404 returned error can't find the container with id 85f223d37af74ec3d26239a3e78c684cb710c49516fb1343fb65971c8b0536a9 Oct 30 20:21:20.641878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.641854 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:20.643693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.643643 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:20.643825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.643737 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:20.643825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.643785 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:20.835720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.835280 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:21:20.840635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.836078 4524 cpu_manager.go:414] "containerMap: removing container" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:20.840635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.836097 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:20.840635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.836259 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3f3e93-8820-4edc-a483-fff28bb89580" containerName="istio-proxy" Oct 30 20:21:20.840635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.836272 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="844d0230-0706-4c0e-a80c-1133fdc14d35" containerName="istio-proxy" Oct 30 20:21:20.840635 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.840099 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:20.856971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.856944 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:20.893848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.893822 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a2386c\"" Oct 30 20:21:20.911608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.911543 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:21:20.935244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.935217 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rjt\" (UniqueName: \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:20.935374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:20.935274 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.043776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.039796 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rjt\" (UniqueName: \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.043776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.039892 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.055721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.050017 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.096339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.096308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:21.113732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.113649 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rjt\" (UniqueName: \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\") pod \"payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.165795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.162554 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:21.214219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.214182 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" event={"ID":"60983cc5-207e-4829-9473-2d91764280c1","Type":"ContainerStarted","Data":"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb"} Oct 30 20:21:21.214374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.214235 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" event={"ID":"60983cc5-207e-4829-9473-2d91764280c1","Type":"ContainerStarted","Data":"85f223d37af74ec3d26239a3e78c684cb710c49516fb1343fb65971c8b0536a9"} Oct 30 20:21:21.340198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.340134 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podStartSLOduration=2.340113798 podStartE2EDuration="2.340113798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:21.337271322 +0000 UTC m=+2376.180942952" watchObservedRunningTime="2026-10-30 20:21:21.340113798 +0000 UTC m=+2376.183785429" Oct 30 20:21:21.580713 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:21.580665 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:21:21.583492 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:21:21.583463 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b6ca18_647d_4394_a749_10fba8eb4be3.slice/crio-6751d3e715438b738385a02fc70dfdf5dc63eda75451715865a36f7d35ad49e5 WatchSource:0}: Error finding container 6751d3e715438b738385a02fc70dfdf5dc63eda75451715865a36f7d35ad49e5: Status 404 returned error can't find the container with id 6751d3e715438b738385a02fc70dfdf5dc63eda75451715865a36f7d35ad49e5 Oct 30 20:21:22.189811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.189774 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:22.191325 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.191301 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:22.191436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.191346 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:22.221760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.221726 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" event={"ID":"d3b6ca18-647d-4394-a749-10fba8eb4be3","Type":"ContainerStarted","Data":"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba"} Oct 30 20:21:22.221887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.221783 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" event={"ID":"d3b6ca18-647d-4394-a749-10fba8eb4be3","Type":"ContainerStarted","Data":"6751d3e715438b738385a02fc70dfdf5dc63eda75451715865a36f7d35ad49e5"} Oct 30 20:21:22.221887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.221811 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:22.339656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.339605 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" podStartSLOduration=2.339590179 podStartE2EDuration="2.339590179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:22.337667832 +0000 UTC m=+2377.181339458" watchObservedRunningTime="2026-10-30 20:21:22.339590179 +0000 UTC m=+2377.183261807" Oct 30 20:21:22.587318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.587239 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:22.932833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:22.932798 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:23.190162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:23.190075 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:23.190527 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:23.190163 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:24.189958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:24.189922 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:24.190140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:24.189985 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:25.189497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:25.189464 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:25.189901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:25.189520 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:25.592772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:25.586308 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:26.189529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:26.189491 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:26.190038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:26.189552 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:27.189933 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:27.189888 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:21:27.190321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:27.189957 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 20:21:28.194608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.194580 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:28.195078 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.195056 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:28.195919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.195897 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:21:28.484790 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.478065 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:28.693068 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.693037 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:28.742846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.742757 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:21:28.743740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.743363 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" containerID="cri-o://e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0" gracePeriod=30 Oct 30 20:21:28.939246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:28.937774 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:29.340958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:29.340910 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:30.725631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:30.725589 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:33.231517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:33.231490 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:21:34.026482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.026414 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:21:34.147813 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.147774 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.147843 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.147893 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148101 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148170 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148217 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148105 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:34.148384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148295 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:34.148436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148390 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148429 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.148535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148483 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:34.148535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.148506 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\") pod \"92e6774b-46b8-49da-af50-2706f3717e27\" (UID: \"92e6774b-46b8-49da-af50-2706f3717e27\") " Oct 30 20:21:34.149095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.149068 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.149235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.149100 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.149235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.149118 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.149357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.149341 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:34.150357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.150327 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:34.150602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.150575 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:21:34.150602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.150592 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "kube-api-access-qrf98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:34.150831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.150811 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:34.151665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.151645 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token" pod "92e6774b-46b8-49da-af50-2706f3717e27" (UID: "92e6774b-46b8-49da-af50-2706f3717e27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:34.250339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250305 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.250339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250335 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e6774b-46b8-49da-af50-2706f3717e27-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.250339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250346 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.251047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250357 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e6774b-46b8-49da-af50-2706f3717e27-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.251047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250366 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrf98\" (UniqueName: \"kubernetes.io/projected/92e6774b-46b8-49da-af50-2706f3717e27-kube-api-access-qrf98\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.251047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.250378 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e6774b-46b8-49da-af50-2706f3717e27-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:34.308092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.308006 4524 generic.go:356] "Generic (PLEG): container finished" podID="92e6774b-46b8-49da-af50-2706f3717e27" containerID="e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0" exitCode=0 Oct 30 20:21:34.308092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.308062 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" Oct 30 20:21:34.308283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.308084 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerDied","Data":"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0"} Oct 30 20:21:34.308283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.308143 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h" event={"ID":"92e6774b-46b8-49da-af50-2706f3717e27","Type":"ContainerDied","Data":"f665cc1332cb2637e1756551b60ab8715d7c374e82aab97101394dd927a94c7f"} Oct 30 20:21:34.308283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.308165 4524 scope.go:122] "RemoveContainer" containerID="e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0" Oct 30 20:21:34.319205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.319186 4524 scope.go:122] "RemoveContainer" containerID="3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c" Oct 30 20:21:34.329986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.329969 4524 scope.go:122] "RemoveContainer" containerID="e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0" Oct 30 20:21:34.330290 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:34.330270 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0\": container with ID starting with e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0 not found: ID does not exist" containerID="e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0" Oct 30 20:21:34.330346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.330299 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0"} err="failed to get container status \"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0\": rpc error: code = NotFound desc = could not find container \"e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0\": container with ID starting with e2393f5cb53032d3ee329877dd50ef58050333cc52501635ada219710b9393f0 not found: ID does not exist" Oct 30 20:21:34.330346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.330317 4524 scope.go:122] "RemoveContainer" containerID="3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c" Oct 30 20:21:34.330568 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:34.330551 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c\": container with ID starting with 3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c not found: ID does not exist" containerID="3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c" Oct 30 20:21:34.330613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.330576 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c"} err="failed to get container status \"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c\": rpc error: code = NotFound desc = could not find container \"3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c\": container with ID starting with 3afc50e44adaa008a52aa6b50b5c9e6352fb9118acfcf91fe384d92d21063b7c not found: ID does not exist" Oct 30 20:21:34.478856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.478824 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:21:34.482443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.482418 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a2103a-openshift-default-cb4dfcb89-rfb8h"] Oct 30 20:21:34.519950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:34.519898 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.002711 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.002678 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:21:35.003362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.003336 4524 cpu_manager.go:414] "containerMap: removing container" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.003362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.003359 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.003592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.003521 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.003592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.003536 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.008342 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.008321 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060023 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060102 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060212 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060243 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060263 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060344 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060386 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060370 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060465 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060408 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.060551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.060486 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlxl\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.075753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.075663 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:21:35.162101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162068 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162105 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162131 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162190 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlxl\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162235 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162310 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162435 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162489 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162517 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162577 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162557 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.162976 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162951 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.163059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.162974 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.163203 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.163178 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.163357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.163335 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.164568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.164548 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.165082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.165066 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.223780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.223749 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.223914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.223891 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlxl\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.302327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.302261 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e6774b-46b8-49da-af50-2706f3717e27" path="/var/lib/kubelet/pods/92e6774b-46b8-49da-af50-2706f3717e27/volumes" Oct 30 20:21:35.322209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.322182 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:35.768825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.768802 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:21:35.770559 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:21:35.770523 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa34f13a_8014_4b62_a7a6_9ce267765bec.slice/crio-2071832b7aa33a2566e06bd0c462a0eee69e6308d02d8532be9fe7fd1482b45a WatchSource:0}: Error finding container 2071832b7aa33a2566e06bd0c462a0eee69e6308d02d8532be9fe7fd1482b45a: Status 404 returned error can't find the container with id 2071832b7aa33a2566e06bd0c462a0eee69e6308d02d8532be9fe7fd1482b45a Oct 30 20:21:35.773463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.773423 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:35.773546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.773512 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:35.773589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.773554 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:35.977217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.977182 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9161-119dfb"] Oct 30 20:21:35.978086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.978066 4524 cpu_manager.go:414] "containerMap: removing container" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.978153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.978089 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e6774b-46b8-49da-af50-2706f3717e27" containerName="istio-proxy" Oct 30 20:21:35.982243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:35.982220 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:36.035160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.035086 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nts8c\"" Oct 30 20:21:36.035160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.035099 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9161-119dfb"] Oct 30 20:21:36.076779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.076745 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s682g\" (UniqueName: \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\") pod \"test-curl-9161-119dfb\" (UID: \"305505d1-5d34-41ca-80ab-0a1d873767c5\") " pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:36.151722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.150391 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:36.178583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.178545 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s682g\" (UniqueName: \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\") pod \"test-curl-9161-119dfb\" (UID: \"305505d1-5d34-41ca-80ab-0a1d873767c5\") " pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:36.241322 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.241297 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s682g\" (UniqueName: \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\") pod \"test-curl-9161-119dfb\" (UID: \"305505d1-5d34-41ca-80ab-0a1d873767c5\") " pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:36.298094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.298024 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:36.322882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.322852 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" event={"ID":"fa34f13a-8014-4b62-a7a6-9ce267765bec","Type":"ContainerStarted","Data":"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585"} Oct 30 20:21:36.323313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.322913 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" event={"ID":"fa34f13a-8014-4b62-a7a6-9ce267765bec","Type":"ContainerStarted","Data":"2071832b7aa33a2566e06bd0c462a0eee69e6308d02d8532be9fe7fd1482b45a"} Oct 30 20:21:36.449876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.449821 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podStartSLOduration=2.449805419 podStartE2EDuration="2.449805419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:36.448550487 +0000 UTC m=+2391.292222142" watchObservedRunningTime="2026-10-30 20:21:36.449805419 +0000 UTC m=+2391.293477048" Oct 30 20:21:36.677995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:36.677967 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9161-119dfb"] Oct 30 20:21:36.679976 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:21:36.679943 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod305505d1_5d34_41ca_80ab_0a1d873767c5.slice/crio-253267341c6c96817f1b2b4d4181b1592f641029608315a014b578f7553173f3 WatchSource:0}: Error finding container 253267341c6c96817f1b2b4d4181b1592f641029608315a014b578f7553173f3: Status 404 returned error can't find the container with id 253267341c6c96817f1b2b4d4181b1592f641029608315a014b578f7553173f3 Oct 30 20:21:37.323032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.322995 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:37.324557 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.324529 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:37.324689 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.324576 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:37.329677 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.329650 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9161-119dfb" event={"ID":"305505d1-5d34-41ca-80ab-0a1d873767c5","Type":"ContainerStarted","Data":"a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86"} Oct 30 20:21:37.329795 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.329687 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9161-119dfb" event={"ID":"305505d1-5d34-41ca-80ab-0a1d873767c5","Type":"ContainerStarted","Data":"253267341c6c96817f1b2b4d4181b1592f641029608315a014b578f7553173f3"} Oct 30 20:21:37.444844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:37.444787 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9161-119dfb" podStartSLOduration=2.444772424 podStartE2EDuration="2.444772424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:37.443462394 +0000 UTC m=+2392.287134024" watchObservedRunningTime="2026-10-30 20:21:37.444772424 +0000 UTC m=+2392.288444053" Oct 30 20:21:38.323402 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:38.323367 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:38.323929 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:38.323421 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:39.009653 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.009618 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9161-119dfb"] Oct 30 20:21:39.013042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.012956 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9161-119dfb"] Oct 30 20:21:39.322900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.322814 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:39.322900 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.322875 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:39.341284 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.341250 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9161-119dfb" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" containerName="test-curl-9161-119dfb" containerID="cri-o://a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86" gracePeriod=2 Oct 30 20:21:39.679340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.679286 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:39.902927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:39.902896 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:40.322808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:40.322775 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:40.322990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:40.322825 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:41.323738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.323671 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:41.338079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.338041 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:41.355334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.355294 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:41.583779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.575190 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:41.817866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.817836 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:41.908390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.908359 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\" (UniqueName: \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\") pod \"305505d1-5d34-41ca-80ab-0a1d873767c5\" (UID: \"305505d1-5d34-41ca-80ab-0a1d873767c5\") " Oct 30 20:21:41.910557 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:41.910530 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g" pod "305505d1-5d34-41ca-80ab-0a1d873767c5" (UID: "305505d1-5d34-41ca-80ab-0a1d873767c5"). InnerVolumeSpecName "kube-api-access-s682g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:42.010107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.010076 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s682g\" (UniqueName: \"kubernetes.io/projected/305505d1-5d34-41ca-80ab-0a1d873767c5-kube-api-access-s682g\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.323018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.322925 4524 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 20:21:42.323018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.322990 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 20:21:42.397380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.395606 4524 generic.go:356] "Generic (PLEG): container finished" podID="305505d1-5d34-41ca-80ab-0a1d873767c5" containerID="a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86" exitCode=137 Oct 30 20:21:42.397380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.395783 4524 scope.go:122] "RemoveContainer" containerID="a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86" Oct 30 20:21:42.397380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.395923 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:42.425176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.425148 4524 scope.go:122] "RemoveContainer" containerID="a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86" Oct 30 20:21:42.427013 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:42.426373 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86\": container with ID starting with a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86 not found: ID does not exist" containerID="a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86" Oct 30 20:21:42.427013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:42.426415 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86"} err="failed to get container status \"a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86\": rpc error: code = NotFound desc = could not find container \"a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86\": container with ID starting with a7fa45e74df58194da70261d5e03c0c6610ced6dee0480f33b964775594d3d86 not found: ID does not exist" Oct 30 20:21:42.483728 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:42.472753 4524 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9161-119dfb\" is forbidden: User \"system:node:ip-10-0-130-41.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-41.us-west-1.compute.internal' and this object" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" pod="opendatahub/test-curl-9161-119dfb" Oct 30 20:21:42.867111 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:42.867066 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:21:43.302844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:43.302806 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" path="/var/lib/kubelet/pods/305505d1-5d34-41ca-80ab-0a1d873767c5/volumes" Oct 30 20:21:43.364734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:43.361986 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:43.364734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:43.362782 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:43.364734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:43.363675 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:21:44.626882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:44.626846 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.425576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.425544 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:21:46.426112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.426098 4524 cpu_manager.go:414] "containerMap: removing container" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" containerName="test-curl-9161-119dfb" Oct 30 20:21:46.426171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.426113 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" containerName="test-curl-9161-119dfb" Oct 30 20:21:46.426238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.426228 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="305505d1-5d34-41ca-80ab-0a1d873767c5" containerName="test-curl-9161-119dfb" Oct 30 20:21:46.430949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.430927 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.484918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.484892 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-01e8dc-serving-cert\"" Oct 30 20:21:46.507913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.507885 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:21:46.539113 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.539080 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.539285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.539207 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.539379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.539361 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mb2\" (UniqueName: \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.539439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.539425 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.640554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.640521 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.640775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.640578 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.640775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.640661 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mb2\" (UniqueName: \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.640775 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:46.640687 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-serving-cert: secret "maas-api-e2e-inf-a-01e8dc-serving-cert" not found Oct 30 20:21:46.641214 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:46.640782 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls podName:c0c8ea09-5ca6-412f-bda8-63a5e68a1a25 nodeName:}" failed. No retries permitted until 2026-10-30 20:21:47.140759484 +0000 UTC m=+2401.984431110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls") pod "maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" (UID: "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25") : secret "maas-api-e2e-inf-a-01e8dc-serving-cert" not found Oct 30 20:21:46.641214 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.640861 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.641335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.641313 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.643242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.643221 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.713768 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.708363 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.721637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.720353 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mb2\" (UniqueName: \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:46.931169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:46.931130 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:47.146982 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:47.146951 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:47.149376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:47.149354 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") pod \"maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:47.160285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:47.160267 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:47.345308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:47.345278 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:47.727067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:47.727039 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:21:47.728581 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:21:47.728552 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c8ea09_5ca6_412f_bda8_63a5e68a1a25.slice/crio-d9d9cba52bc07929ba831fb08232eb003d0ead06f8e83f64cd1d511a86043a82 WatchSource:0}: Error finding container d9d9cba52bc07929ba831fb08232eb003d0ead06f8e83f64cd1d511a86043a82: Status 404 returned error can't find the container with id d9d9cba52bc07929ba831fb08232eb003d0ead06f8e83f64cd1d511a86043a82 Oct 30 20:21:48.079377 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:48.079291 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:48.441400 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:48.441362 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" event={"ID":"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25","Type":"ContainerStarted","Data":"d9d9cba52bc07929ba831fb08232eb003d0ead06f8e83f64cd1d511a86043a82"} Oct 30 20:21:49.449052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:49.449008 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" event={"ID":"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25","Type":"ContainerStarted","Data":"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672"} Oct 30 20:21:49.449545 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:49.449130 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:49.574315 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:49.574255 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" podStartSLOduration=2.643473897 podStartE2EDuration="3.57423653s" totalImagesPullingTime="930.762633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:49.571510998 +0000 UTC m=+2404.415182628" watchObservedRunningTime="2026-10-30 20:21:49.57423653 +0000 UTC m=+2404.417908162" Oct 30 20:21:51.793332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:51.793294 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:52.016752 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:52.016683 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:53.783344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:53.783306 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:21:53.899173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:53.899136 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:21:54.225231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:54.225185 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.449988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:54.449954 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:55.463083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:55.463050 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:21:55.999082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:55.999049 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 20:21:55.999305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:55.999269 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" podUID="22242480-dae8-4d0a-9d77-69742ef38d8e" containerName="maas-api" containerID="cri-o://c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66" gracePeriod=30 Oct 30 20:21:56.173881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.173848 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:56.333597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.333573 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 20:21:56.466746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.462872 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\") pod \"22242480-dae8-4d0a-9d77-69742ef38d8e\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " Oct 30 20:21:56.466746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.462967 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") pod \"22242480-dae8-4d0a-9d77-69742ef38d8e\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " Oct 30 20:21:56.466746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.463013 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\") pod \"22242480-dae8-4d0a-9d77-69742ef38d8e\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " Oct 30 20:21:56.466746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.463216 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\" (UniqueName: \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\") pod \"22242480-dae8-4d0a-9d77-69742ef38d8e\" (UID: \"22242480-dae8-4d0a-9d77-69742ef38d8e\") " Oct 30 20:21:56.470156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.470107 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca" pod "22242480-dae8-4d0a-9d77-69742ef38d8e" (UID: "22242480-dae8-4d0a-9d77-69742ef38d8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:56.475103 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.475035 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8" pod "22242480-dae8-4d0a-9d77-69742ef38d8e" (UID: "22242480-dae8-4d0a-9d77-69742ef38d8e"). InnerVolumeSpecName "kube-api-access-fgtc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:56.489521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.489284 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs" pod "22242480-dae8-4d0a-9d77-69742ef38d8e" (UID: "22242480-dae8-4d0a-9d77-69742ef38d8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:56.490026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.489998 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls" pod "22242480-dae8-4d0a-9d77-69742ef38d8e" (UID: "22242480-dae8-4d0a-9d77-69742ef38d8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:56.542063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.542024 4524 generic.go:356] "Generic (PLEG): container finished" podID="22242480-dae8-4d0a-9d77-69742ef38d8e" containerID="c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66" exitCode=0 Oct 30 20:21:56.542252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.542083 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" event={"ID":"22242480-dae8-4d0a-9d77-69742ef38d8e","Type":"ContainerDied","Data":"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66"} Oct 30 20:21:56.542252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.542155 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" event={"ID":"22242480-dae8-4d0a-9d77-69742ef38d8e","Type":"ContainerDied","Data":"f0aca8867a68a4fa1d1f2f3800711e03a2c684289da98e1340a9c6fa9b1e1e96"} Oct 30 20:21:56.542252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.542087 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw" Oct 30 20:21:56.542252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.542181 4524 scope.go:122] "RemoveContainer" containerID="c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66" Oct 30 20:21:56.552523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.552506 4524 scope.go:122] "RemoveContainer" containerID="c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66" Oct 30 20:21:56.552837 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:56.552817 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66\": container with ID starting with c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66 not found: ID does not exist" containerID="c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66" Oct 30 20:21:56.552901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.552844 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66"} err="failed to get container status \"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66\": rpc error: code = NotFound desc = could not find container \"c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66\": container with ID starting with c2fcdb4e3976bb767ce2aafbbfc254df98d9699b41c6ee8b5b40d5394da89d66 not found: ID does not exist" Oct 30 20:21:56.570618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.570482 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtc8\" (UniqueName: \"kubernetes.io/projected/22242480-dae8-4d0a-9d77-69742ef38d8e-kube-api-access-fgtc8\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.570618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.570515 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.570618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.570530 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22242480-dae8-4d0a-9d77-69742ef38d8e-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.570618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.570544 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22242480-dae8-4d0a-9d77-69742ef38d8e-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.713028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.712994 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 20:21:56.717671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:56.717647 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f84d85-7d866c876b-qbkvw"] Oct 30 20:21:57.300940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:57.300908 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22242480-dae8-4d0a-9d77-69742ef38d8e" path="/var/lib/kubelet/pods/22242480-dae8-4d0a-9d77-69742ef38d8e/volumes" Oct 30 20:21:57.433559 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:57.433529 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:57.651321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:57.651288 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:57.872857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:57.872823 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:58.115127 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:58.115018 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:58.709615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:58.709581 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:58.784104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:58.784073 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" podUID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" containerName="payload-processing" containerID="cri-o://ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009" gracePeriod=25 Oct 30 20:21:58.900998 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:58.900960 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" podUID="49bcec45-1480-41cd-aca1-ca4705351e92" containerName="payload-pre-processing" containerID="cri-o://f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6" gracePeriod=25 Oct 30 20:21:59.064610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.064579 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:21:59.112845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.112806 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\" (UniqueName: \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\") pod \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " Oct 30 20:21:59.113006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.112869 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\") pod \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\" (UID: \"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f\") " Oct 30 20:21:59.113948 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.113919 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume" pod "dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" (UID: "dafc2647-ac35-4bb3-a0ff-6ebc1cba326f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:59.116950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.116908 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck" pod "dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" (UID: "dafc2647-ac35-4bb3-a0ff-6ebc1cba326f"). InnerVolumeSpecName "kube-api-access-k84ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:59.182585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.182561 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:21:59.214985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.214904 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\") pod \"49bcec45-1480-41cd-aca1-ca4705351e92\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " Oct 30 20:21:59.215123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.215004 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\" (UniqueName: \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\") pod \"49bcec45-1480-41cd-aca1-ca4705351e92\" (UID: \"49bcec45-1480-41cd-aca1-ca4705351e92\") " Oct 30 20:21:59.215487 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.215447 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume" pod "49bcec45-1480-41cd-aca1-ca4705351e92" (UID: "49bcec45-1480-41cd-aca1-ca4705351e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:59.216090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.215749 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84ck\" (UniqueName: \"kubernetes.io/projected/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-kube-api-access-k84ck\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:59.216090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.215776 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:59.216090 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.215792 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bcec45-1480-41cd-aca1-ca4705351e92-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:59.217800 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.217773 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg" pod "49bcec45-1480-41cd-aca1-ca4705351e92" (UID: "49bcec45-1480-41cd-aca1-ca4705351e92"). InnerVolumeSpecName "kube-api-access-gdwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:59.317318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.317291 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwxg\" (UniqueName: \"kubernetes.io/projected/49bcec45-1480-41cd-aca1-ca4705351e92-kube-api-access-gdwxg\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:59.563399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.563312 4524 generic.go:356] "Generic (PLEG): container finished" podID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" containerID="ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009" exitCode=0 Oct 30 20:21:59.563399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.563354 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" Oct 30 20:21:59.563631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.563398 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" event={"ID":"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f","Type":"ContainerDied","Data":"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009"} Oct 30 20:21:59.563631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.563448 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw" event={"ID":"dafc2647-ac35-4bb3-a0ff-6ebc1cba326f","Type":"ContainerDied","Data":"4a5dc2196b0bf939dfeaba843ba4a7d00e2172ee1eb747fec02e239999da422b"} Oct 30 20:21:59.563631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.563475 4524 scope.go:122] "RemoveContainer" containerID="ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009" Oct 30 20:21:59.565165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.565137 4524 generic.go:356] "Generic (PLEG): container finished" podID="49bcec45-1480-41cd-aca1-ca4705351e92" containerID="f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6" exitCode=0 Oct 30 20:21:59.565306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.565190 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" Oct 30 20:21:59.565306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.565223 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" event={"ID":"49bcec45-1480-41cd-aca1-ca4705351e92","Type":"ContainerDied","Data":"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6"} Oct 30 20:21:59.565306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.565254 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6" event={"ID":"49bcec45-1480-41cd-aca1-ca4705351e92","Type":"ContainerDied","Data":"56482691f3ceba5951ddcaf7f427a1998893ff0b7663f81c13ce5dc2bcd8d2a5"} Oct 30 20:21:59.573607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.573590 4524 scope.go:122] "RemoveContainer" containerID="ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009" Oct 30 20:21:59.573915 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:59.573896 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009\": container with ID starting with ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009 not found: ID does not exist" containerID="ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009" Oct 30 20:21:59.573967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.573926 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009"} err="failed to get container status \"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009\": rpc error: code = NotFound desc = could not find container \"ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009\": container with ID starting with ec4eb2ba03fd72984d1e91c5ee58c7b2e9d7f55478fd717f9d9105b51f826009 not found: ID does not exist" Oct 30 20:21:59.573967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.573949 4524 scope.go:122] "RemoveContainer" containerID="f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6" Oct 30 20:21:59.582266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.582245 4524 scope.go:122] "RemoveContainer" containerID="f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6" Oct 30 20:21:59.582549 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:21:59.582530 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6\": container with ID starting with f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6 not found: ID does not exist" containerID="f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6" Oct 30 20:21:59.582616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.582558 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6"} err="failed to get container status \"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6\": rpc error: code = NotFound desc = could not find container \"f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6\": container with ID starting with f627372cce821536d558cd64b41378d5ae84c9a02cae76c1b9db6fb267017cf6 not found: ID does not exist" Oct 30 20:21:59.736814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.736788 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:21:59.739257 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.739239 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-2njzw"] Oct 30 20:21:59.851615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.851550 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:21:59.856732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:21:59.856709 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-37732de7-6cf7cf6679-q6vv6"] Oct 30 20:22:01.309215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:01.309182 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49bcec45-1480-41cd-aca1-ca4705351e92" path="/var/lib/kubelet/pods/49bcec45-1480-41cd-aca1-ca4705351e92/volumes" Oct 30 20:22:01.309604 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:01.309588 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" path="/var/lib/kubelet/pods/dafc2647-ac35-4bb3-a0ff-6ebc1cba326f/volumes" Oct 30 20:22:03.255886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:03.253884 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.235421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:06.235385 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.456663 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:06.456625 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.640658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:06.640632 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:22:06.640903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:06.640882 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" containerID="cri-o://058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9" gracePeriod=30 Oct 30 20:22:06.696943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:06.696881 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:07.898822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:07.898787 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:07.907958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:07.907927 4524 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:22:07.908105 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:07.907997 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:22:08.112887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.112812 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:22:08.113435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113418 4524 cpu_manager.go:414] "containerMap: removing container" podUID="22242480-dae8-4d0a-9d77-69742ef38d8e" containerName="maas-api" Oct 30 20:22:08.113508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113437 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22242480-dae8-4d0a-9d77-69742ef38d8e" containerName="maas-api" Oct 30 20:22:08.113508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113463 4524 cpu_manager.go:414] "containerMap: removing container" podUID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" containerName="payload-processing" Oct 30 20:22:08.113508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113472 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" containerName="payload-processing" Oct 30 20:22:08.113508 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113504 4524 cpu_manager.go:414] "containerMap: removing container" podUID="49bcec45-1480-41cd-aca1-ca4705351e92" containerName="payload-pre-processing" Oct 30 20:22:08.113637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113509 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49bcec45-1480-41cd-aca1-ca4705351e92" containerName="payload-pre-processing" Oct 30 20:22:08.113637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113605 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="22242480-dae8-4d0a-9d77-69742ef38d8e" containerName="maas-api" Oct 30 20:22:08.113637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113615 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafc2647-ac35-4bb3-a0ff-6ebc1cba326f" containerName="payload-processing" Oct 30 20:22:08.113637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.113625 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="49bcec45-1480-41cd-aca1-ca4705351e92" containerName="payload-pre-processing" Oct 30 20:22:08.118567 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.118546 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.222499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.222471 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a2386c-serving-cert\"" Oct 30 20:22:08.225346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.225324 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:22:08.242816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.242784 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.242954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.242908 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.243104 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.243083 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.243168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.243138 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w998x\" (UniqueName: \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.344629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.344586 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.344886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.344653 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w998x\" (UniqueName: \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.344886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.344743 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.344886 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.344838 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.345100 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:08.345074 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-serving-cert: secret "maas-api-e2e-inf-b-a2386c-serving-cert" not found Oct 30 20:22:08.345166 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:08.345158 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls podName:28a9f0ac-f3aa-417a-9545-0ed992a1bfa8 nodeName:}" failed. No retries permitted until 2026-10-30 20:22:08.845136869 +0000 UTC m=+2423.688808486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls") pod "maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" (UID: "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8") : secret "maas-api-e2e-inf-b-a2386c-serving-cert" not found Oct 30 20:22:08.346012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.345984 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.355002 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.354979 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.370523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.370461 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:08.455892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.454279 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w998x\" (UniqueName: \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.561721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.537988 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:22:08.594652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.594615 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:08.600218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.598495 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:22:08.600218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.598630 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.670246 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-14c846-openshift-default-dockercfg-thtc5\"" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671452 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rf9\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671514 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671556 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671667 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671720 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671789 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671851 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671888 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.672154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.671937 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773335 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773424 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773490 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773530 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773575 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773652 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rf9\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773718 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773759 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.773989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.773884 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.774172 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.774112 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.774443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.774413 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.774561 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.774508 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.774676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.774639 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.774766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.774736 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.776242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.776218 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.776606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.776588 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.834569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.834536 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.835096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.835071 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rf9\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\") pod \"e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:08.862905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.862876 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:08.874897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.874872 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:08.877127 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:08.877098 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:09.031983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.031895 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:09.043370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.043346 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:09.071139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.071106 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:09.475279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.475248 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:22:09.475551 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:22:09.475510 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a9f0ac_f3aa_417a_9545_0ed992a1bfa8.slice/crio-f50b2e74db6ce5834385d678030a9a5e61bb115711175c315620fbe38b804bde WatchSource:0}: Error finding container f50b2e74db6ce5834385d678030a9a5e61bb115711175c315620fbe38b804bde: Status 404 returned error can't find the container with id f50b2e74db6ce5834385d678030a9a5e61bb115711175c315620fbe38b804bde Oct 30 20:22:09.476726 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:22:09.476684 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8bf821_f4ce_453c_934e_4d9a6b9fbe88.slice/crio-353e8a9ff87e3619fb1bb976eec52d53dcf4a23efe27f5faa80713aad3eab490 WatchSource:0}: Error finding container 353e8a9ff87e3619fb1bb976eec52d53dcf4a23efe27f5faa80713aad3eab490: Status 404 returned error can't find the container with id 353e8a9ff87e3619fb1bb976eec52d53dcf4a23efe27f5faa80713aad3eab490 Oct 30 20:22:09.479992 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.479957 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:09.480093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.480025 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:09.480093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.480060 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:09.523357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.523325 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:22:09.745450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.745349 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" event={"ID":"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8","Type":"ContainerStarted","Data":"f50b2e74db6ce5834385d678030a9a5e61bb115711175c315620fbe38b804bde"} Oct 30 20:22:09.747770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.747734 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerStarted","Data":"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004"} Oct 30 20:22:09.747991 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.747796 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerStarted","Data":"353e8a9ff87e3619fb1bb976eec52d53dcf4a23efe27f5faa80713aad3eab490"} Oct 30 20:22:09.871301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:09.871245 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podStartSLOduration=1.871229124 podStartE2EDuration="1.871229124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:09.868452484 +0000 UTC m=+2424.712124115" watchObservedRunningTime="2026-10-30 20:22:09.871229124 +0000 UTC m=+2424.714900753" Oct 30 20:22:10.047221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.047071 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:10.056426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.056392 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:10.765044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.764995 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" event={"ID":"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8","Type":"ContainerStarted","Data":"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056"} Oct 30 20:22:10.765402 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.765382 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:10.765489 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.765433 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:10.766587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.766563 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:22:10.907724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:10.907604 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" podStartSLOduration=2.035128486 podStartE2EDuration="2.907586318s" totalImagesPullingTime="872.457832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:10.906480881 +0000 UTC m=+2425.750152510" watchObservedRunningTime="2026-10-30 20:22:10.907586318 +0000 UTC m=+2425.751257948" Oct 30 20:22:11.971669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:11.971643 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:22:12.145959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.145914 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.145969 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146004 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146037 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146129 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146166 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146225 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146249 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.146384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146274 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\") pod \"2c00b976-7d95-4eb0-b308-0dc214523fc3\" (UID: \"2c00b976-7d95-4eb0-b308-0dc214523fc3\") " Oct 30 20:22:12.147218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.146839 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:12.147218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.147015 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:12.147218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.147192 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:12.147506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.147477 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:12.152794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.149419 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "kube-api-access-g8nz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:12.152794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.149456 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:12.152794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.149677 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:12.153111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.153086 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:12.154073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.154039 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert" pod "2c00b976-7d95-4eb0-b308-0dc214523fc3" (UID: "2c00b976-7d95-4eb0-b308-0dc214523fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:12.247878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247802 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.247878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247836 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.247878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247851 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.247878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247865 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c00b976-7d95-4eb0-b308-0dc214523fc3-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.247878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247879 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.248251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247892 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8nz5\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-kube-api-access-g8nz5\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.248251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247906 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.248251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247920 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.248251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.247932 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c00b976-7d95-4eb0-b308-0dc214523fc3-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:12.627459 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.624604 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:12.788196 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:12.788158 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:22:12.799510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.799480 4524 generic.go:356] "Generic (PLEG): container finished" podID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerID="058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9" exitCode=0 Oct 30 20:22:12.799672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.799526 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" Oct 30 20:22:12.799672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.799559 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" event={"ID":"2c00b976-7d95-4eb0-b308-0dc214523fc3","Type":"ContainerDied","Data":"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9"} Oct 30 20:22:12.799672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.799609 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4" event={"ID":"2c00b976-7d95-4eb0-b308-0dc214523fc3","Type":"ContainerDied","Data":"4936ca8abca9230786816127b39d709fa3a941539eda788b1a541bbc29d709dd"} Oct 30 20:22:12.799672 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.799630 4524 scope.go:122] "RemoveContainer" containerID="058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9" Oct 30 20:22:12.809891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.809874 4524 scope.go:122] "RemoveContainer" containerID="058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9" Oct 30 20:22:12.810173 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:12.810157 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9\": container with ID starting with 058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9 not found: ID does not exist" containerID="058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9" Oct 30 20:22:12.810239 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.810179 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9"} err="failed to get container status \"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9\": rpc error: code = NotFound desc = could not find container \"058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9\": container with ID starting with 058f6c808325c49789fd6ba661e0fc61155f86d6757e2ba1b0e7627c2dc663c9 not found: ID does not exist" Oct 30 20:22:12.862990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.862829 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:12.994684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:12.994648 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:22:13.006939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:13.006908 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-jbtp4"] Oct 30 20:22:13.111784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:13.111750 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.379770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:13.378986 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" path="/var/lib/kubelet/pods/2c00b976-7d95-4eb0-b308-0dc214523fc3/volumes" Oct 30 20:22:13.888470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:13.888424 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:14.778384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:14.778353 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:16.799397 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:16.799368 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:22:17.419154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:17.419120 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:17.680791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:17.677322 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.461141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.461108 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:22:18.461589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.461345 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" podUID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" containerName="maas-api" containerID="cri-o://ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b" gracePeriod=30 Oct 30 20:22:18.506950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.506918 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.766680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.765310 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.809919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.809892 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:22:18.909153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909120 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\" (UniqueName: \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\") pod \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " Oct 30 20:22:18.909297 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909192 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\") pod \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " Oct 30 20:22:18.909349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909321 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") pod \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " Oct 30 20:22:18.909422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909404 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\") pod \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\" (UID: \"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1\") " Oct 30 20:22:18.909868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909807 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca" pod "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" (UID: "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:18.909985 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.909941 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:18.911107 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.911080 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp" pod "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" (UID: "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1"). InnerVolumeSpecName "kube-api-access-m47zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:18.911239 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.911221 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs" pod "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" (UID: "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:18.911382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:18.911357 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls" pod "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" (UID: "02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.015810 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.015846 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47zp\" (UniqueName: \"kubernetes.io/projected/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-kube-api-access-m47zp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.015862 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.020632 4524 generic.go:356] "Generic (PLEG): container finished" podID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" containerID="ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b" exitCode=0 Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.020841 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" event={"ID":"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1","Type":"ContainerDied","Data":"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b"} Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.020931 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" event={"ID":"02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1","Type":"ContainerDied","Data":"80971d7ddff14937b53d5c25a245eeb56938df75028e9c9e9927324843b6e383"} Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.020959 4524 scope.go:122] "RemoveContainer" containerID="ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b" Oct 30 20:22:19.036067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.021100 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2" Oct 30 20:22:19.061402 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.061370 4524 scope.go:122] "RemoveContainer" containerID="ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b" Oct 30 20:22:19.062268 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:19.062184 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b\": container with ID starting with ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b not found: ID does not exist" containerID="ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b" Oct 30 20:22:19.062268 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.062228 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b"} err="failed to get container status \"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b\": rpc error: code = NotFound desc = could not find container \"ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b\": container with ID starting with ee82ca18d8a867806e66b6867bca5f1b820304ce52764a06c813b0262a38c69b not found: ID does not exist" Oct 30 20:22:19.209289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.209232 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:22:19.217302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.212295 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37732de7-7b44f7679-cgbr2"] Oct 30 20:22:19.301252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:19.301170 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" path="/var/lib/kubelet/pods/02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1/volumes" Oct 30 20:22:20.410599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:20.410556 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:20.658725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:20.651352 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:20.862222 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:20.862129 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:21.424376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:21.424345 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:25.847960 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:25.846635 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:26.458361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:26.458326 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:31.168212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:31.168181 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:31.880094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:31.880061 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:36.260834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:36.260673 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.238466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:41.238422 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:42.291725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:42.291192 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:42.759251 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:42.759212 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:42.800716 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:22:42.800657 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:22:43.294422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.294388 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:22:43.296119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.296091 4524 cpu_manager.go:414] "containerMap: removing container" podUID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" containerName="maas-api" Oct 30 20:22:43.296290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.296274 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" containerName="maas-api" Oct 30 20:22:43.296417 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.296404 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" Oct 30 20:22:43.297191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.297170 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" Oct 30 20:22:43.297583 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.297515 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cd6e01-20ed-46eb-8bc6-cccf6bd3ebf1" containerName="maas-api" Oct 30 20:22:43.297747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.297731 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c00b976-7d95-4eb0-b308-0dc214523fc3" containerName="istio-proxy" Oct 30 20:22:43.332046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.318913 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.380717 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.361551 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.381027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.380992 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.399846 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400073 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400102 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfnp\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400146 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400227 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400401 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.406756 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.400431 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.495754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.473062 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-453277-openshift-default-dockercfg-wqfgr\"" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501469 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501522 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501556 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501676 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501717 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfnp\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501755 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501790 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501917 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.510447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.501953 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.512643 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.512354 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:22:43.605807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.596605 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.635360 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.596652 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.635595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.596858 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.635709 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.596970 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.635818 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.610363 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.635928 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.614038 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.636039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.632692 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.637161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.637092 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.638383 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.638365 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfnp\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.640723 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.640063 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\") pod \"e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:43.856363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.856274 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.915066 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:43.915031 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:44.382482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:44.382445 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:22:44.421179 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:22:44.421138 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463488c8_7f01_4773_a3c8_72dc66418507.slice/crio-8383803b0438edf93b20cd870bc9aa2b41782635d4ab6537378bed013cc678ee WatchSource:0}: Error finding container 8383803b0438edf93b20cd870bc9aa2b41782635d4ab6537378bed013cc678ee: Status 404 returned error can't find the container with id 8383803b0438edf93b20cd870bc9aa2b41782635d4ab6537378bed013cc678ee Oct 30 20:22:44.423950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:44.423915 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:44.424067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:44.424000 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:44.424067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:44.424044 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:45.229889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:45.229836 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerStarted","Data":"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf"} Oct 30 20:22:45.230101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:45.229914 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerStarted","Data":"8383803b0438edf93b20cd870bc9aa2b41782635d4ab6537378bed013cc678ee"} Oct 30 20:22:45.403827 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:45.403772 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podStartSLOduration=2.4037538720000002 podStartE2EDuration="2.403753872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:45.400622967 +0000 UTC m=+2460.244294595" watchObservedRunningTime="2026-10-30 20:22:45.403753872 +0000 UTC m=+2460.247425506" Oct 30 20:22:45.915801 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:45.915768 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:45.921506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:45.921480 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:46.236839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:46.236752 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:46.238040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:46.238012 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:22:46.901053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:46.901016 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:47.991918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:47.991886 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.235240 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:48.235206 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.480362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:48.480332 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:49.070050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:49.069985 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:49.287427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:49.287397 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:50.263491 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.263457 4524 generic.go:356] "Generic (PLEG): container finished" podID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerID="d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554" exitCode=0 Oct 30 20:22:50.263946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.263514 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerDied","Data":"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554"} Oct 30 20:22:50.263946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.263932 4524 scope.go:122] "RemoveContainer" containerID="d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554" Oct 30 20:22:50.265460 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.265434 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:50.265531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.265500 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:50.265575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:50.265531 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:22:51.291443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:51.291406 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerStarted","Data":"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8"} Oct 30 20:22:51.771721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:51.771190 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:22:51.773288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:51.773256 4524 patch_prober.go:29] interesting pod/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:22:51.773531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:51.773504 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:22:52.469156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:52.469115 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:52.692765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:52.686023 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:52.773507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:52.773419 4524 patch_prober.go:29] interesting pod/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:22:52.773858 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:52.773822 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:22:53.730461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:53.730424 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:22:53.772189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:53.772145 4524 patch_prober.go:29] interesting pod/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:22:53.772395 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:53.772222 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:22:54.121455 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:54.121381 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:22:54.121617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:54.121594 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" containerID="cri-o://935df5f8a573428b015bd860263454994c87d8e14d9aa1c589d9bfe2650a7877" gracePeriod=30 Oct 30 20:22:54.772153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:54.772113 4524 patch_prober.go:29] interesting pod/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:22:54.772861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:54.772825 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:22:55.779285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:55.779249 4524 patch_prober.go:29] interesting pod/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:22:55.779866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:55.779836 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:22:56.776139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:56.776058 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:22:56.777138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:56.777117 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:22:57.713059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:57.710854 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:59.352941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.352912 4524 generic.go:356] "Generic (PLEG): container finished" podID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerID="935df5f8a573428b015bd860263454994c87d8e14d9aa1c589d9bfe2650a7877" exitCode=0 Oct 30 20:22:59.353448 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.352960 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" event={"ID":"da4f63b0-12dc-4476-a0a6-b53b975721ee","Type":"ContainerDied","Data":"935df5f8a573428b015bd860263454994c87d8e14d9aa1c589d9bfe2650a7877"} Oct 30 20:22:59.409600 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.409564 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:22:59.471966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.471934 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472031 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472067 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472117 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472153 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472224 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472510 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472230 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:59.472589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472427 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:59.472589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472446 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:59.472589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472532 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:59.472589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472567 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.472874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.472775 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\") pod \"da4f63b0-12dc-4476-a0a6-b53b975721ee\" (UID: \"da4f63b0-12dc-4476-a0a6-b53b975721ee\") " Oct 30 20:22:59.473319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.473276 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:59.473437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.473393 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.473498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.473433 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.473498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.473450 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4f63b0-12dc-4476-a0a6-b53b975721ee-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.473498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.473466 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.474873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.474848 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:59.475022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.474993 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:59.475145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.475021 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:59.475145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.475103 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp" pod "da4f63b0-12dc-4476-a0a6-b53b975721ee" (UID: "da4f63b0-12dc-4476-a0a6-b53b975721ee"). InnerVolumeSpecName "kube-api-access-hc6xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:59.574308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.574274 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.574308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.574299 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc6xp\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-kube-api-access-hc6xp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.574308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.574310 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.574574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.574319 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:59.574574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:22:59.574327 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4f63b0-12dc-4476-a0a6-b53b975721ee-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:00.362574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:00.362534 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" event={"ID":"da4f63b0-12dc-4476-a0a6-b53b975721ee","Type":"ContainerDied","Data":"7a73da447ca612d7cdbfae70cdb5dc349b07bd0c69dcd4d7aca635ff9a2fa79f"} Oct 30 20:23:00.362574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:00.362564 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n" Oct 30 20:23:00.363159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:00.362611 4524 scope.go:122] "RemoveContainer" containerID="935df5f8a573428b015bd860263454994c87d8e14d9aa1c589d9bfe2650a7877" Oct 30 20:23:00.581156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:00.581114 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:23:00.629479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:00.629406 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5lg8n"] Oct 30 20:23:01.299922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:01.299883 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" path="/var/lib/kubelet/pods/da4f63b0-12dc-4476-a0a6-b53b975721ee/volumes" Oct 30 20:23:02.529142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.529103 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:23:02.529714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.529689 4524 cpu_manager.go:414] "containerMap: removing container" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" Oct 30 20:23:02.529771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.529715 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" Oct 30 20:23:02.529830 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.529821 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4f63b0-12dc-4476-a0a6-b53b975721ee" containerName="istio-proxy" Oct 30 20:23:02.534974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.534950 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.588869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.588841 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-14c846-serving-cert\"" Oct 30 20:23:02.620747 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.620689 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.620966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.620941 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.621082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.621058 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnh8q\" (UniqueName: \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.621132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.621108 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.642036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.642009 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:23:02.722335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.722303 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.722512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.722357 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnh8q\" (UniqueName: \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.722512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.722385 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.722512 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:02.722454 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-serving-cert: secret "maas-api-e2e-inf-a-14c846-serving-cert" not found Oct 30 20:23:02.722512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.722478 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.722512 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:02.722514 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls podName:a2547fe3-68f3-4141-88b5-091263f916db nodeName:}" failed. No retries permitted until 2026-10-30 20:23:03.222498102 +0000 UTC m=+2478.066169723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls") pod "maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" (UID: "a2547fe3-68f3-4141-88b5-091263f916db") : secret "maas-api-e2e-inf-a-14c846-serving-cert" not found Oct 30 20:23:02.723184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.723150 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.724906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.724888 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.804808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.804668 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnh8q\" (UniqueName: \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:02.809782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:02.809747 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.027295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.027261 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.229761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.229724 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:03.232333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.232311 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") pod \"maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:03.267427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.267400 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.447608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.447576 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:03.838308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:03.838284 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:23:03.838743 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:23:03.838389 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2547fe3_68f3_4141_88b5_091263f916db.slice/crio-b842008004929debe220535f1fb412fa18784bb69062aeb6e15ae50964f45327 WatchSource:0}: Error finding container b842008004929debe220535f1fb412fa18784bb69062aeb6e15ae50964f45327: Status 404 returned error can't find the container with id b842008004929debe220535f1fb412fa18784bb69062aeb6e15ae50964f45327 Oct 30 20:23:04.021300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:04.014834 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:04.387911 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:04.387877 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" event={"ID":"a2547fe3-68f3-4141-88b5-091263f916db","Type":"ContainerStarted","Data":"b842008004929debe220535f1fb412fa18784bb69062aeb6e15ae50964f45327"} Oct 30 20:23:04.496493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:04.496458 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:04.979393 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:04.979361 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:05.395002 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:05.394960 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" event={"ID":"a2547fe3-68f3-4141-88b5-091263f916db","Type":"ContainerStarted","Data":"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c"} Oct 30 20:23:05.395180 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:05.395032 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:05.516811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:05.516755 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" podStartSLOduration=2.469446424 podStartE2EDuration="3.516738855s" totalImagesPullingTime="1.047292431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:05.514943319 +0000 UTC m=+2480.358614951" watchObservedRunningTime="2026-10-30 20:23:05.516738855 +0000 UTC m=+2480.360410483" Oct 30 20:23:07.374608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.374575 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 20:23:07.375030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.374868 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" podUID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" containerName="maas-api" containerID="cri-o://afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c" gracePeriod=30 Oct 30 20:23:07.543319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.543283 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:07.704086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.703787 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 20:23:07.814788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.805878 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\") pod \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " Oct 30 20:23:07.814788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.805959 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\") pod \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " Oct 30 20:23:07.814788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.805994 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\" (UniqueName: \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\") pod \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " Oct 30 20:23:07.814788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.806084 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") pod \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\" (UID: \"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1\") " Oct 30 20:23:07.814788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.809536 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca" pod "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" (UID: "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:07.834409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.827996 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp" pod "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" (UID: "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1"). InnerVolumeSpecName "kube-api-access-b97dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:07.836983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.836925 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls" pod "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" (UID: "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:07.851028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.850347 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs" pod "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" (UID: "6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:07.908857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.908809 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:07.908857 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.908852 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97dp\" (UniqueName: \"kubernetes.io/projected/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-kube-api-access-b97dp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:07.909120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.908871 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:07.909120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:07.908886 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:08.415597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.415563 4524 generic.go:356] "Generic (PLEG): container finished" podID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" containerID="afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c" exitCode=0 Oct 30 20:23:08.416129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.415612 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" event={"ID":"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1","Type":"ContainerDied","Data":"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c"} Oct 30 20:23:08.416129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.415619 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" Oct 30 20:23:08.416129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.415650 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw" event={"ID":"6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1","Type":"ContainerDied","Data":"4e3f7043e69ec25c180b9ad9e7e4668fe25f6eab1ceaf987665c5fe42ad2842f"} Oct 30 20:23:08.416129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.415673 4524 scope.go:122] "RemoveContainer" containerID="afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c" Oct 30 20:23:08.426313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.426295 4524 scope.go:122] "RemoveContainer" containerID="afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c" Oct 30 20:23:08.426581 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:08.426560 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c\": container with ID starting with afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c not found: ID does not exist" containerID="afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c" Oct 30 20:23:08.426678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.426595 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c"} err="failed to get container status \"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c\": rpc error: code = NotFound desc = could not find container \"afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c\": container with ID starting with afff7455ae49d7e6a8108370397f79faa91af0bad8877f93b7e3c619fb26483c not found: ID does not exist" Oct 30 20:23:08.599129 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.599094 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 20:23:08.648792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:08.648756 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d67ea9-6d4bb89bbb-g22bw"] Oct 30 20:23:09.238446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:09.238412 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:09.300347 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:09.300311 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" path="/var/lib/kubelet/pods/6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1/volumes" Oct 30 20:23:09.496091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:09.496014 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:09.752295 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:09.752215 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:10.216264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:10.216233 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:11.407066 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:11.407021 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:23:12.276813 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:12.276779 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:12.786091 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:12.786055 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:23:14.675384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:14.675351 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:14.894552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:14.894521 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.236371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:17.233210 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.499958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:17.499823 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.710622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:17.710585 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.955722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:17.950218 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:19.592472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:19.592433 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:23:19.789280 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:19.789244 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.094721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.092356 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.417738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.417318 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.637578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.637550 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.721625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.721548 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:23:20.721797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.721761 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" containerID="cri-o://f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437" gracePeriod=30 Oct 30 20:23:20.758575 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.758535 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:23:20.759060 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.759029 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" containerID="cri-o://ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb" gracePeriod=30 Oct 30 20:23:20.771828 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.771805 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:23:20.772044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.772021 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" containerID="cri-o://cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585" gracePeriod=30 Oct 30 20:23:20.927481 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:20.927391 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.327477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:25.327437 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:26.129506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.129470 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:23:26.203237 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.203211 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:23:26.209873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.209821 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:23:26.231463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231437 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.231595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231481 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.231595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231523 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.231595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231560 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.231595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231588 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231622 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231653 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231681 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231745 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231781 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.231892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231827 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231867 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231979 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.231988 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232086 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232145 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232170 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.232217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232201 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232216 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232224 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232292 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232388 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232387 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232430 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232471 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\") pod \"85e0f1dd-5699-4b55-bf54-d248a80c132c\" (UID: \"85e0f1dd-5699-4b55-bf54-d248a80c132c\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232490 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232508 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.232569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232568 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.233110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232652 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\") pod \"60983cc5-207e-4829-9473-2d91764280c1\" (UID: \"60983cc5-207e-4829-9473-2d91764280c1\") " Oct 30 20:23:26.233110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232739 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.233110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232796 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\") pod \"fa34f13a-8014-4b62-a7a6-9ce267765bec\" (UID: \"fa34f13a-8014-4b62-a7a6-9ce267765bec\") " Oct 30 20:23:26.233411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233381 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa34f13a-8014-4b62-a7a6-9ce267765bec-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.233490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233410 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.233490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233426 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.233490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233440 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.235993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232496 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.232509 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233332 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233555 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233647 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233760 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.233929 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.235472 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.235564 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.235765 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.236122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.235967 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.236271 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.236808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.236784 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.237772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.237717 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:26.237772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.237734 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.238169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.238138 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:26.238169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.238142 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "kube-api-access-pgbhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.238524 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.238501 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.238658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.238639 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:26.239053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.239026 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "kube-api-access-489cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.239292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.239274 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl" pod "fa34f13a-8014-4b62-a7a6-9ce267765bec" (UID: "fa34f13a-8014-4b62-a7a6-9ce267765bec"). InnerVolumeSpecName "kube-api-access-7hlxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.240045 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.240026 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo" pod "60983cc5-207e-4829-9473-2d91764280c1" (UID: "60983cc5-207e-4829-9473-2d91764280c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:26.240592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.240559 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token" pod "85e0f1dd-5699-4b55-bf54-d248a80c132c" (UID: "85e0f1dd-5699-4b55-bf54-d248a80c132c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334534 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334564 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334576 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334585 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334594 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e0f1dd-5699-4b55-bf54-d248a80c132c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334603 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlxl\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-kube-api-access-7hlxl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.334611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334612 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-489cg\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-kube-api-access-489cg\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334621 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334630 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334638 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60983cc5-207e-4829-9473-2d91764280c1-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334646 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334654 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334662 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334670 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334680 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334691 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbhr\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-kube-api-access-pgbhr\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334727 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334741 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa34f13a-8014-4b62-a7a6-9ce267765bec-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334754 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334767 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60983cc5-207e-4829-9473-2d91764280c1-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334779 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60983cc5-207e-4829-9473-2d91764280c1-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334788 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60983cc5-207e-4829-9473-2d91764280c1-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.335187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.334796 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e0f1dd-5699-4b55-bf54-d248a80c132c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:26.599154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.599067 4524 generic.go:356] "Generic (PLEG): container finished" podID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerID="cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585" exitCode=0 Oct 30 20:23:26.599154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.599132 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" Oct 30 20:23:26.599380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.599172 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" event={"ID":"fa34f13a-8014-4b62-a7a6-9ce267765bec","Type":"ContainerDied","Data":"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585"} Oct 30 20:23:26.599380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.599252 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh" event={"ID":"fa34f13a-8014-4b62-a7a6-9ce267765bec","Type":"ContainerDied","Data":"2071832b7aa33a2566e06bd0c462a0eee69e6308d02d8532be9fe7fd1482b45a"} Oct 30 20:23:26.599380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.599285 4524 scope.go:122] "RemoveContainer" containerID="cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585" Oct 30 20:23:26.600944 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.600918 4524 generic.go:356] "Generic (PLEG): container finished" podID="60983cc5-207e-4829-9473-2d91764280c1" containerID="ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb" exitCode=0 Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.601038 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.601285 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" event={"ID":"60983cc5-207e-4829-9473-2d91764280c1","Type":"ContainerDied","Data":"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb"} Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.601354 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp" event={"ID":"60983cc5-207e-4829-9473-2d91764280c1","Type":"ContainerDied","Data":"85f223d37af74ec3d26239a3e78c684cb710c49516fb1343fb65971c8b0536a9"} Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.602772 4524 generic.go:356] "Generic (PLEG): container finished" podID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerID="f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437" exitCode=0 Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.602807 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" event={"ID":"85e0f1dd-5699-4b55-bf54-d248a80c132c","Type":"ContainerDied","Data":"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437"} Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.602829 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" Oct 30 20:23:26.604744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.602835 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8" event={"ID":"85e0f1dd-5699-4b55-bf54-d248a80c132c","Type":"ContainerDied","Data":"20e49d40d081ce7206e82830eef6a3c8ddb378cefb62ec5fead82e615a5477f0"} Oct 30 20:23:26.618779 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.618481 4524 scope.go:122] "RemoveContainer" containerID="cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585" Oct 30 20:23:26.623454 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:26.620724 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585\": container with ID starting with cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585 not found: ID does not exist" containerID="cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585" Oct 30 20:23:26.623454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.620783 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585"} err="failed to get container status \"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585\": rpc error: code = NotFound desc = could not find container \"cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585\": container with ID starting with cf3a2d5ca614d036afb09a079f1d619cb9f0f536c2ac701654a3a3e6794df585 not found: ID does not exist" Oct 30 20:23:26.623454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.620806 4524 scope.go:122] "RemoveContainer" containerID="ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb" Oct 30 20:23:26.637243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.637227 4524 scope.go:122] "RemoveContainer" containerID="ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb" Oct 30 20:23:26.637501 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:26.637480 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb\": container with ID starting with ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb not found: ID does not exist" containerID="ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb" Oct 30 20:23:26.637555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.637510 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb"} err="failed to get container status \"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb\": rpc error: code = NotFound desc = could not find container \"ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb\": container with ID starting with ea0db8074a6f1d7d1efa0238ab3606ede0f491e5b3dacf55427c14ed250ca9cb not found: ID does not exist" Oct 30 20:23:26.637555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.637527 4524 scope.go:122] "RemoveContainer" containerID="f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437" Oct 30 20:23:26.679218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.679001 4524 scope.go:122] "RemoveContainer" containerID="f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437" Oct 30 20:23:26.679453 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:26.679428 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437\": container with ID starting with f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437 not found: ID does not exist" containerID="f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437" Oct 30 20:23:26.679548 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.679464 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437"} err="failed to get container status \"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437\": rpc error: code = NotFound desc = could not find container \"f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437\": container with ID starting with f89878be6b0fc1324354f379bc9e3bdebd1ed1a0dc2793b6e33f31225aa69437 not found: ID does not exist" Oct 30 20:23:26.781744 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.781679 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:23:26.784195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.784153 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-7lfjp"] Oct 30 20:23:26.899582 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.899550 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:23:26.902052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:26.902031 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-p7ws8"] Oct 30 20:23:27.015124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.015092 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:23:27.019571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.019546 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-zpfjh"] Oct 30 20:23:27.121733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.118515 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:27.328772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.326485 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60983cc5-207e-4829-9473-2d91764280c1" path="/var/lib/kubelet/pods/60983cc5-207e-4829-9473-2d91764280c1/volumes" Oct 30 20:23:27.328772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.327265 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" path="/var/lib/kubelet/pods/85e0f1dd-5699-4b55-bf54-d248a80c132c/volumes" Oct 30 20:23:27.328772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.328027 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" path="/var/lib/kubelet/pods/fa34f13a-8014-4b62-a7a6-9ce267765bec/volumes" Oct 30 20:23:27.332718 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:27.331637 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:30.373123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:30.366968 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:31.641847 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.641811 4524 generic.go:356] "Generic (PLEG): container finished" podID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerID="8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004" exitCode=0 Oct 30 20:23:31.642301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.641880 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerDied","Data":"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004"} Oct 30 20:23:31.642301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.642272 4524 scope.go:122] "RemoveContainer" containerID="8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004" Oct 30 20:23:31.643746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.643694 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:31.643819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.643790 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:31.643856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:31.643821 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:23:32.674612 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:32.674575 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerStarted","Data":"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12"} Oct 30 20:23:33.043788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:33.043681 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:23:33.043952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:33.043888 4524 patch_prober.go:29] interesting pod/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 20:23:33.043952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:33.043929 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 20:23:34.043803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:34.043763 4524 patch_prober.go:29] interesting pod/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 20:23:34.044209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:34.043836 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 20:23:34.613034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:34.612968 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" containerID="cri-o://dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4" gracePeriod=45 Oct 30 20:23:34.928571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:34.928545 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:23:35.044258 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.044224 4524 patch_prober.go:29] interesting pod/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 20:23:35.044822 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.044281 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 20:23:35.082505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082467 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.082690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082533 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\" (UniqueName: \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.082690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082567 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.082690 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082627 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.082909 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082763 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.082909 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082785 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:35.083022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.082925 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\") pod \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\" (UID: \"c4801444-2aa3-4e3b-946e-e1dea3aa32c6\") " Oct 30 20:23:35.083022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.083000 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:35.083531 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.083416 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:35.083646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.083572 4524 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-home\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.083646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.083592 4524 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tmp-dir\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.083646 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.083606 4524 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-model-cache\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.085955 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.085904 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:35.085955 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.085900 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "kube-api-access-kwql2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:35.086219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.086192 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs" pod "c4801444-2aa3-4e3b-946e-e1dea3aa32c6" (UID: "c4801444-2aa3-4e3b-946e-e1dea3aa32c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:35.185141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.185057 4524 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-tls-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.185141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.185089 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwql2\" (UniqueName: \"kubernetes.io/projected/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-kube-api-access-kwql2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.185141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.185102 4524 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4801444-2aa3-4e3b-946e-e1dea3aa32c6-dshm\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:35.270721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.270501 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:35.692728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.692679 4524 generic.go:356] "Generic (PLEG): container finished" podID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerID="dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4" exitCode=0 Oct 30 20:23:35.692957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.692733 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" Oct 30 20:23:35.692957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.692730 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" event={"ID":"c4801444-2aa3-4e3b-946e-e1dea3aa32c6","Type":"ContainerDied","Data":"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4"} Oct 30 20:23:35.692957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.692789 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm" event={"ID":"c4801444-2aa3-4e3b-946e-e1dea3aa32c6","Type":"ContainerDied","Data":"8ca4a42bdd6465e81a001b1d8b9615f36c5f98f5e890ca818845a218bfbf43da"} Oct 30 20:23:35.692957 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.692813 4524 scope.go:122] "RemoveContainer" containerID="dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4" Oct 30 20:23:35.702187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.702167 4524 scope.go:122] "RemoveContainer" containerID="dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4" Oct 30 20:23:35.702477 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:35.702455 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4\": container with ID starting with dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4 not found: ID does not exist" containerID="dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4" Oct 30 20:23:35.702549 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.702486 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4"} err="failed to get container status \"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4\": rpc error: code = NotFound desc = could not find container \"dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4\": container with ID starting with dbc3a4eecba6a24c727da7009ea5f4ce09962f3fda5541b3225e31d661e51db4 not found: ID does not exist" Oct 30 20:23:35.872197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.872160 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:23:35.874589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:35.874566 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/premium-w4-bebb42-kserve-75c6fcdd7f-xq9qm"] Oct 30 20:23:36.044544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:36.044458 4524 patch_prober.go:29] interesting pod/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 20:23:36.044544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:36.044510 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 20:23:37.044131 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:37.044099 4524 patch_prober.go:29] interesting pod/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 20:23:37.044312 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:37.044154 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 20:23:37.300939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:37.300861 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" path="/var/lib/kubelet/pods/c4801444-2aa3-4e3b-946e-e1dea3aa32c6/volumes" Oct 30 20:23:38.048799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:38.048771 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:23:38.049084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:38.049065 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:23:38.049616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:38.049599 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:23:38.794765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:38.794731 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:40.303462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:40.303432 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:40.550360 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:40.550331 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:40.758616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:40.758587 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:42.788514 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:23:42.788476 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:23:43.670638 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:43.670600 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:23:43.857422 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:43.857385 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:45.787424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:45.787386 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:48.846384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:48.846344 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:50.728961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:50.728929 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:51.302884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:51.302843 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:51.510122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:51.510090 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:55.587191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:55.587150 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:56.462793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:56.462670 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:56.681731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:56.681679 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:57.528821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:57.528766 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:58.704912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:58.704871 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" containerID="cri-o://db1dcaeb44ecc000e6ec7baeec5d6c189d17ca86cccdb5045240732ab4cfa5cb" gracePeriod=45 Oct 30 20:23:58.854362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:58.854331 4524 generic.go:356] "Generic (PLEG): container finished" podID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerID="db1dcaeb44ecc000e6ec7baeec5d6c189d17ca86cccdb5045240732ab4cfa5cb" exitCode=0 Oct 30 20:23:58.854542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:58.854402 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" event={"ID":"7589ded7-4ca6-4489-803b-2dd4ec79bc12","Type":"ContainerDied","Data":"db1dcaeb44ecc000e6ec7baeec5d6c189d17ca86cccdb5045240732ab4cfa5cb"} Oct 30 20:23:58.951142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:58.951115 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:58.963589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:58.963534 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:23:59.099153 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099115 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\" (UniqueName: \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099338 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099242 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099337 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099365 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099458 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099495 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\" (UniqueName: \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\") pod \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\" (UID: \"7589ded7-4ca6-4489-803b-2dd4ec79bc12\") " Oct 30 20:23:59.099529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099496 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:59.099716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099625 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:59.099785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.099748 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:59.100095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.100072 4524 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-model-cache\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.100207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.100102 4524 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tmp-dir\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.100207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.100117 4524 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-home\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.102189 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.102162 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:59.102306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.102237 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "kube-api-access-zj7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:59.102480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.102452 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs" pod "7589ded7-4ca6-4489-803b-2dd4ec79bc12" (UID: "7589ded7-4ca6-4489-803b-2dd4ec79bc12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:59.201191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.201158 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7nw\" (UniqueName: \"kubernetes.io/projected/7589ded7-4ca6-4489-803b-2dd4ec79bc12-kube-api-access-zj7nw\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.201191 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.201190 4524 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7589ded7-4ca6-4489-803b-2dd4ec79bc12-dshm\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.201439 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.201201 4524 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7589ded7-4ca6-4489-803b-2dd4ec79bc12-tls-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:23:59.862526 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.862491 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" event={"ID":"7589ded7-4ca6-4489-803b-2dd4ec79bc12","Type":"ContainerDied","Data":"dc16026e1e20386cb6f8cb3c05c5a2af41bb4e50a0a435dd9bc6db8506302353"} Oct 30 20:23:59.863017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.862545 4524 scope.go:122] "RemoveContainer" containerID="db1dcaeb44ecc000e6ec7baeec5d6c189d17ca86cccdb5045240732ab4cfa5cb" Oct 30 20:23:59.863017 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:23:59.862508 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2" Oct 30 20:24:00.042622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:00.042585 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:24:00.092123 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:00.092085 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-2-w6-f75272-kserve-5c8759578f-pr6r2"] Oct 30 20:24:00.450835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:00.448390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:01.299793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:01.299759 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" path="/var/lib/kubelet/pods/7589ded7-4ca6-4489-803b-2dd4ec79bc12/volumes" Oct 30 20:24:03.466990 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:03.466953 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:05.773833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:05.773802 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:05.991346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:05.991316 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:10.719482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:10.719449 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:11.156234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:11.146794 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:11.357652 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:11.357622 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:12.150495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:12.150458 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:12.393062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:12.393032 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:12.786342 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:24:12.786299 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:24:16.205429 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:16.205391 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:16.882586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:16.882549 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:17.167599 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:17.167567 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:20.442355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:20.442280 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:20.657139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:20.657106 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:21.007110 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007078 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:24:21.007595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007582 4524 cpu_manager.go:414] "containerMap: removing container" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007598 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007611 4524 cpu_manager.go:414] "containerMap: removing container" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007616 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007631 4524 cpu_manager.go:414] "containerMap: removing container" podUID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" containerName="maas-api" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007636 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" containerName="maas-api" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007643 4524 cpu_manager.go:414] "containerMap: removing container" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" Oct 30 20:24:21.007659 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007648 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007670 4524 cpu_manager.go:414] "containerMap: removing container" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007675 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007682 4524 cpu_manager.go:414] "containerMap: removing container" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007687 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007795 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="7589ded7-4ca6-4489-803b-2dd4ec79bc12" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007804 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4801444-2aa3-4e3b-946e-e1dea3aa32c6" containerName="main" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007813 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa34f13a-8014-4b62-a7a6-9ce267765bec" containerName="istio-proxy" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007819 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e0f1dd-5699-4b55-bf54-d248a80c132c" containerName="istio-proxy" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007825 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc92a9d-b2a0-4cd9-bfb2-1a702fc66fd1" containerName="maas-api" Oct 30 20:24:21.007889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.007829 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="60983cc5-207e-4829-9473-2d91764280c1" containerName="istio-proxy" Oct 30 20:24:21.014831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.014272 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.070070 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.070034 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:24:21.115830 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.115785 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.116033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.115987 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.116224 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.116197 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-72db69-dockercfg-hxrm4\"" Oct 30 20:24:21.116302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.116283 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-72db69\"" Oct 30 20:24:21.217682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.217647 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.217850 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.217831 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.218507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.218489 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.279300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.279218 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\") pod \"payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.331883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.331852 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:21.710806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.710775 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:24:21.712403 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:24:21.712364 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbe29f2_115b_4722_a144_fa1c981cd0a4.slice/crio-3df2917f2144b9a593c186d100a6665f80306b15e68f221206ffcc72c6c8c75f WatchSource:0}: Error finding container 3df2917f2144b9a593c186d100a6665f80306b15e68f221206ffcc72c6c8c75f: Status 404 returned error can't find the container with id 3df2917f2144b9a593c186d100a6665f80306b15e68f221206ffcc72c6c8c75f Oct 30 20:24:21.853101 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:21.851188 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:22.031833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.030939 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" event={"ID":"4fbe29f2-115b-4722-a144-fa1c981cd0a4","Type":"ContainerStarted","Data":"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4"} Oct 30 20:24:22.031833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.031009 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" event={"ID":"4fbe29f2-115b-4722-a144-fa1c981cd0a4","Type":"ContainerStarted","Data":"3df2917f2144b9a593c186d100a6665f80306b15e68f221206ffcc72c6c8c75f"} Oct 30 20:24:22.032271 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.032246 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:22.152027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.151476 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" podStartSLOduration=2.151455212 podStartE2EDuration="2.151455212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:22.150527064 +0000 UTC m=+2556.994198693" watchObservedRunningTime="2026-10-30 20:24:22.151455212 +0000 UTC m=+2556.995126842" Oct 30 20:24:22.167320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.167263 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:22.376610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:22.376494 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:23.343433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:23.343393 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:23.568789 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:23.568758 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:25.052406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.052328 4524 generic.go:356] "Generic (PLEG): container finished" podID="463488c8-7f01-4773-a3c8-72dc66418507" containerID="9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf" exitCode=0 Oct 30 20:24:25.052940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.052404 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerDied","Data":"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf"} Oct 30 20:24:25.052940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.052842 4524 scope.go:122] "RemoveContainer" containerID="9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf" Oct 30 20:24:25.054368 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.054336 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:24:25.054476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.054408 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:24:25.054476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:25.054440 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:24:26.086069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.086030 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerStarted","Data":"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a"} Oct 30 20:24:26.591109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.591075 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.841065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.841029 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.915506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.915473 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:24:26.916458 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.916432 4524 patch_prober.go:29] interesting pod/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:24:26.916584 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:26.916493 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:24:27.916208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:27.916169 4524 patch_prober.go:29] interesting pod/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:24:27.916610 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:27.916237 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:24:28.915334 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:28.915301 4524 patch_prober.go:29] interesting pod/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:24:28.915529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:28.915355 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:24:29.915340 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:29.915303 4524 patch_prober.go:29] interesting pod/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:24:29.915760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:29.915358 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:24:29.915987 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:29.915966 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:30.929722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:30.929106 4524 patch_prober.go:29] interesting pod/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:24:30.929722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:30.929176 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:24:31.237544 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:31.237452 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:24:31.681581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:31.681546 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:31.920246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:31.920214 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:24:32.122430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:32.122351 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:24:34.046728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:34.046676 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:24:35.820039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:35.819997 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:36.754682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:36.754642 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:41.957186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:41.955648 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:42.785218 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:24:42.785182 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:24:47.071729 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:47.071679 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:49.118682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:49.118643 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:50.699675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:50.699631 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:51.123361 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.123288 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 20:24:51.123534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.123475 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" podUID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" containerName="maas-api" containerID="cri-o://1c6a808db9a418969d4b7b5d20ce8765de244b95b3668980c4d502702c48c42e" gracePeriod=30 Oct 30 20:24:51.241196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.241145 4524 generic.go:356] "Generic (PLEG): container finished" podID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" containerID="1c6a808db9a418969d4b7b5d20ce8765de244b95b3668980c4d502702c48c42e" exitCode=0 Oct 30 20:24:51.241196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.241181 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" event={"ID":"ee87cffe-c28e-4278-9d4a-087968dd6f4c","Type":"ContainerDied","Data":"1c6a808db9a418969d4b7b5d20ce8765de244b95b3668980c4d502702c48c42e"} Oct 30 20:24:51.285224 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.285174 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:51.454037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.454010 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 20:24:51.547367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.547333 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\" (UniqueName: \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\") pod \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " Oct 30 20:24:51.547585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.547406 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") pod \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " Oct 30 20:24:51.547585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.547442 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\") pod \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " Oct 30 20:24:51.547585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.547504 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\") pod \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\" (UID: \"ee87cffe-c28e-4278-9d4a-087968dd6f4c\") " Oct 30 20:24:51.549662 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.549628 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca" pod "ee87cffe-c28e-4278-9d4a-087968dd6f4c" (UID: "ee87cffe-c28e-4278-9d4a-087968dd6f4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:51.552415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.552372 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs" pod "ee87cffe-c28e-4278-9d4a-087968dd6f4c" (UID: "ee87cffe-c28e-4278-9d4a-087968dd6f4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:51.555128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.555098 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls" pod "ee87cffe-c28e-4278-9d4a-087968dd6f4c" (UID: "ee87cffe-c28e-4278-9d4a-087968dd6f4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:51.557039 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.556994 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl" pod "ee87cffe-c28e-4278-9d4a-087968dd6f4c" (UID: "ee87cffe-c28e-4278-9d4a-087968dd6f4c"). InnerVolumeSpecName "kube-api-access-prprl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:51.649095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.649064 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prprl\" (UniqueName: \"kubernetes.io/projected/ee87cffe-c28e-4278-9d4a-087968dd6f4c-kube-api-access-prprl\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:51.649095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.649094 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:51.649279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.649104 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee87cffe-c28e-4278-9d4a-087968dd6f4c-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:51.649279 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:51.649114 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee87cffe-c28e-4278-9d4a-087968dd6f4c-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:52.249206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.249172 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" Oct 30 20:24:52.249206 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.249181 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj" event={"ID":"ee87cffe-c28e-4278-9d4a-087968dd6f4c","Type":"ContainerDied","Data":"8cf2b596083fe7f448bedb5fa16351a85412e3f8715b186fa7e245d34760ff60"} Oct 30 20:24:52.249773 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.249246 4524 scope.go:122] "RemoveContainer" containerID="1c6a808db9a418969d4b7b5d20ce8765de244b95b3668980c4d502702c48c42e" Oct 30 20:24:52.415097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.415062 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 20:24:52.421793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.421757 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f75272-567fdc5895-pwpsj"] Oct 30 20:24:52.462020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.461971 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:52.863023 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:52.862988 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:53.133371 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:53.133287 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:53.328188 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:53.328133 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" path="/var/lib/kubelet/pods/ee87cffe-c28e-4278-9d4a-087968dd6f4c/volumes" Oct 30 20:24:53.849879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:53.849844 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.062363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:54.062330 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.323421 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:54.323390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.527238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:54.527209 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:55.123516 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:55.123482 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:59.596637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:24:59.596600 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:00.584306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:00.584266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:00.802661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:00.802604 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:01.029289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:01.029256 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:02.534811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:02.534774 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:02.785184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:02.785111 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:02.835301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:02.835270 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 20:25:02.835520 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:02.835498 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" containerID="cri-o://32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6" gracePeriod=30 Oct 30 20:25:03.091758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:03.091643 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:04.160370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.160335 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:25:04.161218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.161198 4524 cpu_manager.go:414] "containerMap: removing container" podUID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" containerName="maas-api" Oct 30 20:25:04.161298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.161220 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" containerName="maas-api" Oct 30 20:25:04.161343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.161322 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee87cffe-c28e-4278-9d4a-087968dd6f4c" containerName="maas-api" Oct 30 20:25:04.166309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.166290 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.220044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.220019 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.220200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.220184 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.220250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.220236 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.220329 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.220294 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4qw\" (UniqueName: \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.220618 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.220595 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-72db69-serving-cert\"" Oct 30 20:25:04.274684 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.274646 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:25:04.322026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.321988 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.322197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.322090 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.322197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.322116 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.322197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.322143 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4qw\" (UniqueName: \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.322381 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:04.322257 4524 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-serving-cert: secret "maas-api-e2e-inf-b-72db69-serving-cert" not found Oct 30 20:25:04.322381 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:04.322332 4524 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls podName:3a33b4c9-1302-47dc-972a-211dd750410b nodeName:}" failed. No retries permitted until 2026-10-30 20:25:04.822310433 +0000 UTC m=+2599.665982059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls") pod "maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" (UID: "3a33b4c9-1302-47dc-972a-211dd750410b") : secret "maas-api-e2e-inf-b-72db69-serving-cert" not found Oct 30 20:25:04.322734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.322695 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.324842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.324817 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.396223 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.396188 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4qw\" (UniqueName: \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.506194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.506113 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:04.773724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.773637 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:04.837276 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.837246 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:04.839669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:04.839643 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") pod \"maas-api-e2e-inf-b-72db69-c55fdc488-pzz58\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:05.081782 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:05.081689 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:05.462001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:05.461972 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:25:05.466856 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:25:05.466829 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a33b4c9_1302_47dc_972a_211dd750410b.slice/crio-8090a0e7011d36975f45f2a62903c11c3d62a9627d8f095eec9f9a9ffd7a7a5d WatchSource:0}: Error finding container 8090a0e7011d36975f45f2a62903c11c3d62a9627d8f095eec9f9a9ffd7a7a5d: Status 404 returned error can't find the container with id 8090a0e7011d36975f45f2a62903c11c3d62a9627d8f095eec9f9a9ffd7a7a5d Oct 30 20:25:05.467983 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:05.467955 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:25:06.359661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:06.359620 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" event={"ID":"3a33b4c9-1302-47dc-972a-211dd750410b","Type":"ContainerStarted","Data":"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6"} Oct 30 20:25:06.359869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:06.359694 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:06.359869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:06.359731 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" event={"ID":"3a33b4c9-1302-47dc-972a-211dd750410b","Type":"ContainerStarted","Data":"8090a0e7011d36975f45f2a62903c11c3d62a9627d8f095eec9f9a9ffd7a7a5d"} Oct 30 20:25:06.479788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:06.479741 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" podStartSLOduration=1.697219788 podStartE2EDuration="2.479727328s" totalImagesPullingTime="782.50754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:06.478491923 +0000 UTC m=+2601.322163551" watchObservedRunningTime="2026-10-30 20:25:06.479727328 +0000 UTC m=+2601.323398953" Oct 30 20:25:08.106053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.106028 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 20:25:08.176028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176003 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176054 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176093 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176154 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176173 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176198 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176237 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:08.176407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176269 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176306 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176407 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176342 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\") pod \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\" (UID: \"3c4bf278-0627-4ef8-b1e0-f101c91f0549\") " Oct 30 20:25:08.176666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176521 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:08.176666 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176574 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:08.176815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176723 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:08.176873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176838 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:08.176873 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176843 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.176953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176891 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.176953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176911 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.176953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.176925 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.178709 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.178677 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:08.178800 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.178769 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "kube-api-access-qclhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:08.179035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.179013 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:08.179084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.179014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo" pod "3c4bf278-0627-4ef8-b1e0-f101c91f0549" (UID: "3c4bf278-0627-4ef8-b1e0-f101c91f0549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:08.278092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.278054 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.278092 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.278087 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclhs\" (UniqueName: \"kubernetes.io/projected/3c4bf278-0627-4ef8-b1e0-f101c91f0549-kube-api-access-qclhs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.278314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.278103 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.278314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.278117 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.278314 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.278138 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4bf278-0627-4ef8-b1e0-f101c91f0549-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:08.375833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.375758 4524 generic.go:356] "Generic (PLEG): container finished" podID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerID="32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6" exitCode=0 Oct 30 20:25:08.375833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.375798 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerDied","Data":"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6"} Oct 30 20:25:08.376038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.375828 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" Oct 30 20:25:08.376038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.375847 4524 scope.go:122] "RemoveContainer" containerID="32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6" Oct 30 20:25:08.376038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.375838 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5" event={"ID":"3c4bf278-0627-4ef8-b1e0-f101c91f0549","Type":"ContainerDied","Data":"0ef6910ede3b0ef885a2a0c9178704c5c9b7e234a86d9e95de78f3929e48fa50"} Oct 30 20:25:08.385815 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.385795 4524 scope.go:122] "RemoveContainer" containerID="45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc" Oct 30 20:25:08.394425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.394404 4524 scope.go:122] "RemoveContainer" containerID="32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6" Oct 30 20:25:08.394714 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:08.394685 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6\": container with ID starting with 32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6 not found: ID does not exist" containerID="32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6" Oct 30 20:25:08.394784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.394723 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6"} err="failed to get container status \"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6\": rpc error: code = NotFound desc = could not find container \"32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6\": container with ID starting with 32988abb1e1b9d4281156a4b2ec9dd9ed627845d79aa3be8539aa0e4e28aaab6 not found: ID does not exist" Oct 30 20:25:08.394784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.394742 4524 scope.go:122] "RemoveContainer" containerID="45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc" Oct 30 20:25:08.394988 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:08.394970 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc\": container with ID starting with 45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc not found: ID does not exist" containerID="45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc" Oct 30 20:25:08.395034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.394993 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc"} err="failed to get container status \"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc\": rpc error: code = NotFound desc = could not find container \"45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc\": container with ID starting with 45854c685ae76b86fa03dc6e99e3aca7c67ed2c21ce5352c64fd48c7e804c7dc not found: ID does not exist" Oct 30 20:25:08.543642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.543610 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 20:25:08.547456 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:08.547430 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-mtnk5"] Oct 30 20:25:09.299346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:09.299313 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" path="/var/lib/kubelet/pods/3c4bf278-0627-4ef8-b1e0-f101c91f0549/volumes" Oct 30 20:25:09.552079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:09.551992 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:12.372869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:12.372838 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:25:12.785492 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:12.785458 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:25:14.336637 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:14.336600 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:15.441200 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:15.441154 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:15.653507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:15.653472 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:18.218175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:18.218141 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:19.159136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:19.158796 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:24.008720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:24.005138 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:28.838375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:28.838340 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:30.021833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:30.021681 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:33.957447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:33.953548 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:38.717571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:38.717536 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:42.787173 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:25:42.787123 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:25:43.540882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:43.540842 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:48.345208 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:48.345177 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:49.480391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:49.480361 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:51.493946 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:51.493914 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:51.747359 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:51.747287 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:53.251030 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:53.250988 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:58.246804 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:25:58.246555 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:00.267059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:00.267026 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:03.326212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:03.326176 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:08.126341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:08.126265 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:10.060665 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:10.060628 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:12.155217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.155179 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:26:12.155943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.155920 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.156050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.155944 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.156050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.155970 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.156050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.155978 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.156211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.156149 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.156211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.156162 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4bf278-0627-4ef8-b1e0-f101c91f0549" containerName="istio-proxy" Oct 30 20:26:12.160986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.160964 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.234871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.234835 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.235037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.234951 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.235037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.234984 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.235119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.235041 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.235119 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.235109 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzgp\" (UniqueName: \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.235265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.235237 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.271720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.271663 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-14c846\"/\"kube-root-ca.crt\"" Oct 30 20:26:12.271966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.271942 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-14c846\"/\"default-dockercfg-jkc7b\"" Oct 30 20:26:12.272049 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.271973 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-14c846\"/\"openshift-service-ca.crt\"" Oct 30 20:26:12.272099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.272059 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-14c846\"/\"test-model-14c846-kserve-self-signed-certs\"" Oct 30 20:26:12.274769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.274743 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:26:12.327275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.327241 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:12.340529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340488 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzgp\" (UniqueName: \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.340936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340570 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.340936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340661 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.340936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340754 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.340936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340793 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.340936 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.340824 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.341374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.341345 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.341977 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.341944 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.342207 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.342185 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.344661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.344641 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.345158 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.345140 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.402889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.402863 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzgp\" (UniqueName: \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\") pod \"test-model-14c846-kserve-6db6fdb5b7-k89rd\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.474012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.473930 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:12.794033 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:12.793942 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:26:12.857503 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.857466 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:26:12.858986 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:26:12.858955 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbce877b5_0218_4c55_8d5d_25083e1894ea.slice/crio-16d85153b46c7afecfe84f687f3f3702341b2e74ca11721452015c477da8d91a WatchSource:0}: Error finding container 16d85153b46c7afecfe84f687f3f3702341b2e74ca11721452015c477da8d91a: Status 404 returned error can't find the container with id 16d85153b46c7afecfe84f687f3f3702341b2e74ca11721452015c477da8d91a Oct 30 20:26:12.900760 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:12.896390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:13.756909 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:13.756876 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" event={"ID":"bce877b5-0218-4c55-8d5d-25083e1894ea","Type":"ContainerStarted","Data":"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba"} Oct 30 20:26:13.756909 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:13.756920 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" event={"ID":"bce877b5-0218-4c55-8d5d-25083e1894ea","Type":"ContainerStarted","Data":"16d85153b46c7afecfe84f687f3f3702341b2e74ca11721452015c477da8d91a"} Oct 30 20:26:13.876130 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:13.876076 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" podStartSLOduration=1.8760620609999998 podStartE2EDuration="1.876062061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:13.874651874 +0000 UTC m=+2668.718323504" watchObservedRunningTime="2026-10-30 20:26:13.876062061 +0000 UTC m=+2668.719733690" Oct 30 20:26:14.474912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:14.474886 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:17.854485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:17.852440 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:22.474148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:22.474111 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:22.487138 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:22.487113 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:22.678299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:22.678266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:22.819787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:22.819687 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:26:27.596335 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:27.596295 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:32.438952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:32.438903 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:37.341908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:37.341872 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:42.189995 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:42.189956 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:42.785636 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:42.785597 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:26:44.248145 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.248112 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:26:44.248579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.248334 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" containerID="cri-o://ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c" gracePeriod=30 Oct 30 20:26:44.298586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.298547 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:26:44.298824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.298799 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" containerID="cri-o://6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6" gracePeriod=30 Oct 30 20:26:44.301076 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.301054 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:26:44.301250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.301227 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" containerID="cri-o://616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d" gracePeriod=30 Oct 30 20:26:44.348307 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.348281 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:26:44.348470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:44.348450 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" containerID="cri-o://35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe" gracePeriod=30 Oct 30 20:26:46.281722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:46.281209 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:47.128842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:47.128803 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:49.572727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.572449 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:26:49.634771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.634743 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:26:49.672217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.672192 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:26:49.677918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.677894 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.677931 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.677959 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678019 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678167 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678188 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.678238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678225 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678410 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678279 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678410 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678306 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678410 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678406 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\") pod \"321a585b-81da-4d31-af5b-4de25af3e1fb\" (UID: \"321a585b-81da-4d31-af5b-4de25af3e1fb\") " Oct 30 20:26:49.678598 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678570 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:49.678986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678961 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.679741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.678991 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321a585b-81da-4d31-af5b-4de25af3e1fb-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.679741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.679074 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.679741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.679168 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.679741 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.679313 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.681399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.681377 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:49.681488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.681398 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "kube-api-access-vmjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.681488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.681455 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.682601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.682584 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy" pod "321a585b-81da-4d31-af5b-4de25af3e1fb" (UID: "321a585b-81da-4d31-af5b-4de25af3e1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.715447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.715423 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:26:49.780375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780343 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780390 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780417 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780455 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780502 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780555 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780585 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780586 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780624 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780658 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780687 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780756 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780811 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780848 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780858 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.780941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780915 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780949 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.780989 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781009 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\") pod \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\" (UID: \"f9c1c650-6d4c-45cb-b5f2-56e523b54365\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781072 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781099 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\") pod \"871b5b61-8f4a-4e16-bff7-f18801b67b21\" (UID: \"871b5b61-8f4a-4e16-bff7-f18801b67b21\") " Oct 30 20:26:49.781346 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781303 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.781761 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781594 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:49.781898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781874 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781904 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjnj\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-kube-api-access-vmjnj\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781919 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781919 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781938 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781952 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781965 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.781989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781979 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.782384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.781992 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.782384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782008 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.782384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782021 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.782384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782034 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321a585b-81da-4d31-af5b-4de25af3e1fb-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.782384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782187 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.782654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782429 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.782757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782607 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:49.782757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782684 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.782907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.782885 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.785142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.785117 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.785907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.785883 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:49.786979 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.786955 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "kube-api-access-npgzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.787062 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.787026 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.787182 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.787162 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.787446 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.787417 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:49.787901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.787881 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj" pod "871b5b61-8f4a-4e16-bff7-f18801b67b21" (UID: "871b5b61-8f4a-4e16-bff7-f18801b67b21"). InnerVolumeSpecName "kube-api-access-qfznj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.788376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.788347 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token" pod "f9c1c650-6d4c-45cb-b5f2-56e523b54365" (UID: "f9c1c650-6d4c-45cb-b5f2-56e523b54365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.838726 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.838693 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.887924 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888001 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888048 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888090 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888162 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888233 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888260 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888321 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888416 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\") pod \"1e262ae1-982f-4de1-9f21-422fa840d6bd\" (UID: \"1e262ae1-982f-4de1-9f21-422fa840d6bd\") " Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888924 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888961 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888976 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.888981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.888992 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfznj\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-kube-api-access-qfznj\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889009 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889023 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871b5b61-8f4a-4e16-bff7-f18801b67b21-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889036 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889050 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889064 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889077 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgzc\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-kube-api-access-npgzc\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889091 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c1c650-6d4c-45cb-b5f2-56e523b54365-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889107 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c1c650-6d4c-45cb-b5f2-56e523b54365-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889122 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871b5b61-8f4a-4e16-bff7-f18801b67b21-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889135 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871b5b61-8f4a-4e16-bff7-f18801b67b21-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889658 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.889939 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.890318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.890183 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.906390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.896077 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "kube-api-access-j2v6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.906390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.900054 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:49.906390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.900368 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.906390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.901213 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:49.908912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.908808 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:49.908912 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.908878 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo" pod "1e262ae1-982f-4de1-9f21-422fa840d6bd" (UID: "1e262ae1-982f-4de1-9f21-422fa840d6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:49.990658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990623 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990656 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990669 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990683 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v6k\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-kube-api-access-j2v6k\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990710 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990723 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990735 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990749 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e262ae1-982f-4de1-9f21-422fa840d6bd-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.990902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.990761 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e262ae1-982f-4de1-9f21-422fa840d6bd-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:26:49.997026 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.996990 4524 generic.go:356] "Generic (PLEG): container finished" podID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerID="ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c" exitCode=0 Oct 30 20:26:49.997185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.997066 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" Oct 30 20:26:49.997185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.997081 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" event={"ID":"321a585b-81da-4d31-af5b-4de25af3e1fb","Type":"ContainerDied","Data":"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c"} Oct 30 20:26:49.997185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.997146 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg" event={"ID":"321a585b-81da-4d31-af5b-4de25af3e1fb","Type":"ContainerDied","Data":"cb1311f2c10d20467d630dbd1d1ee684a4d889bd1b95a6af56c3440240a6a4df"} Oct 30 20:26:49.997185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.997174 4524 scope.go:122] "RemoveContainer" containerID="ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c" Oct 30 20:26:49.998603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.998574 4524 generic.go:356] "Generic (PLEG): container finished" podID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerID="6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6" exitCode=0 Oct 30 20:26:49.998764 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.998683 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" Oct 30 20:26:49.998764 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.998726 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" event={"ID":"1e262ae1-982f-4de1-9f21-422fa840d6bd","Type":"ContainerDied","Data":"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6"} Oct 30 20:26:49.998906 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:49.998763 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx" event={"ID":"1e262ae1-982f-4de1-9f21-422fa840d6bd","Type":"ContainerDied","Data":"6dce1f0d87e81ac829ac08c7020d5044ad381567814c861997bc0745c9cdf190"} Oct 30 20:26:50.002374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.002348 4524 generic.go:356] "Generic (PLEG): container finished" podID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerID="35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe" exitCode=0 Oct 30 20:26:50.002625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.002390 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" Oct 30 20:26:50.002625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.002434 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" event={"ID":"f9c1c650-6d4c-45cb-b5f2-56e523b54365","Type":"ContainerDied","Data":"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe"} Oct 30 20:26:50.002625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.002484 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm" event={"ID":"f9c1c650-6d4c-45cb-b5f2-56e523b54365","Type":"ContainerDied","Data":"cac623b62ea61a76a148d82da1d34e0ca47eea0aaaa9465302fded43b5a22cea"} Oct 30 20:26:50.006171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.006016 4524 generic.go:356] "Generic (PLEG): container finished" podID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerID="616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d" exitCode=0 Oct 30 20:26:50.006171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.006057 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" event={"ID":"871b5b61-8f4a-4e16-bff7-f18801b67b21","Type":"ContainerDied","Data":"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d"} Oct 30 20:26:50.006171 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.006096 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" event={"ID":"871b5b61-8f4a-4e16-bff7-f18801b67b21","Type":"ContainerDied","Data":"a82f8ec114aa49c6dd9cf88c69b35a008dd6876c9edba5ed1ecaa3491e5e3760"} Oct 30 20:26:50.006398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.006174 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r" Oct 30 20:26:50.009943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.009928 4524 scope.go:122] "RemoveContainer" containerID="ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c" Oct 30 20:26:50.010208 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:50.010181 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c\": container with ID starting with ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c not found: ID does not exist" containerID="ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c" Oct 30 20:26:50.010308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.010215 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c"} err="failed to get container status \"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c\": rpc error: code = NotFound desc = could not find container \"ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c\": container with ID starting with ba1593a7f43d7f72e66ac15c9b4ea26564f0fae63f8a5cb65b637a4774e06a4c not found: ID does not exist" Oct 30 20:26:50.010308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.010242 4524 scope.go:122] "RemoveContainer" containerID="6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6" Oct 30 20:26:50.020308 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.020289 4524 scope.go:122] "RemoveContainer" containerID="6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6" Oct 30 20:26:50.020576 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:50.020551 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6\": container with ID starting with 6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6 not found: ID does not exist" containerID="6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6" Oct 30 20:26:50.020642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.020588 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6"} err="failed to get container status \"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6\": rpc error: code = NotFound desc = could not find container \"6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6\": container with ID starting with 6a726e11b60cafbca189a308307e7e191d458d079daae644be30485c247f26f6 not found: ID does not exist" Oct 30 20:26:50.020642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.020615 4524 scope.go:122] "RemoveContainer" containerID="35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe" Oct 30 20:26:50.031963 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.031907 4524 scope.go:122] "RemoveContainer" containerID="35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe" Oct 30 20:26:50.032216 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:50.032186 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe\": container with ID starting with 35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe not found: ID does not exist" containerID="35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe" Oct 30 20:26:50.032300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.032216 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe"} err="failed to get container status \"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe\": rpc error: code = NotFound desc = could not find container \"35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe\": container with ID starting with 35005a1a340ac9e444b0232035319eb910d0aaee7b36d8185c5826b37ac4defe not found: ID does not exist" Oct 30 20:26:50.032300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.032240 4524 scope.go:122] "RemoveContainer" containerID="616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d" Oct 30 20:26:50.043111 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.043088 4524 scope.go:122] "RemoveContainer" containerID="616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d" Oct 30 20:26:50.043466 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:26:50.043439 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d\": container with ID starting with 616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d not found: ID does not exist" containerID="616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d" Oct 30 20:26:50.044233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.044207 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d"} err="failed to get container status \"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d\": rpc error: code = NotFound desc = could not find container \"616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d\": container with ID starting with 616c14670a7f3f7d397d335693c8f51b2f06bd665f143791b91f264e17da9c1d not found: ID does not exist" Oct 30 20:26:50.286720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.286630 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:26:50.290233 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.290210 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t624r"] Oct 30 20:26:50.430737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.426182 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:26:50.430737 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.428530 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2z4mx"] Oct 30 20:26:50.515601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.515571 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:26:50.520608 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.520582 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-v95qm"] Oct 30 20:26:50.630390 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.630317 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:26:50.637010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:50.636985 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-q28fg"] Oct 30 20:26:51.300237 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:51.300196 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" path="/var/lib/kubelet/pods/1e262ae1-982f-4de1-9f21-422fa840d6bd/volumes" Oct 30 20:26:51.300763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:51.300741 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" path="/var/lib/kubelet/pods/321a585b-81da-4d31-af5b-4de25af3e1fb/volumes" Oct 30 20:26:51.301236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:51.301217 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" path="/var/lib/kubelet/pods/871b5b61-8f4a-4e16-bff7-f18801b67b21/volumes" Oct 30 20:26:51.302034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:51.302011 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" path="/var/lib/kubelet/pods/f9c1c650-6d4c-45cb-b5f2-56e523b54365/volumes" Oct 30 20:26:52.219603 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:52.219570 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:57.046327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:57.046278 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:57.694794 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:26:57.693230 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.698496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:00.698467 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:27:00.806420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:00.806388 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:27:01.077534 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.077443 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:27:01.077742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.077714 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" podUID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" containerName="maas-api" containerID="cri-o://77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649" gracePeriod=30 Oct 30 20:27:01.125532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.125495 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:01.348071 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.348043 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:01.367311 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.367292 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:27:01.420621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.420599 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\" (UniqueName: \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\") pod \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " Oct 30 20:27:01.420753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.420661 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\") pod \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " Oct 30 20:27:01.420753 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.420690 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\") pod \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " Oct 30 20:27:01.420880 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.420779 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") pod \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\" (UID: \"6d788ee1-cd0c-486c-9ee4-3bd725f1433b\") " Oct 30 20:27:01.421147 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.421124 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca" pod "6d788ee1-cd0c-486c-9ee4-3bd725f1433b" (UID: "6d788ee1-cd0c-486c-9ee4-3bd725f1433b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:01.422816 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.422788 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs" pod "6d788ee1-cd0c-486c-9ee4-3bd725f1433b" (UID: "6d788ee1-cd0c-486c-9ee4-3bd725f1433b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:01.422892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.422815 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls" pod "6d788ee1-cd0c-486c-9ee4-3bd725f1433b" (UID: "6d788ee1-cd0c-486c-9ee4-3bd725f1433b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:01.422892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.422862 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d" pod "6d788ee1-cd0c-486c-9ee4-3bd725f1433b" (UID: "6d788ee1-cd0c-486c-9ee4-3bd725f1433b"). InnerVolumeSpecName "kube-api-access-2tl2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:01.522607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.522572 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tl2d\" (UniqueName: \"kubernetes.io/projected/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-kube-api-access-2tl2d\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:01.522607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.522607 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:01.522607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.522623 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:01.522952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:01.522637 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d788ee1-cd0c-486c-9ee4-3bd725f1433b-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:02.077427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.077393 4524 generic.go:356] "Generic (PLEG): container finished" podID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" containerID="77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649" exitCode=0 Oct 30 20:27:02.077993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.077440 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" Oct 30 20:27:02.077993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.077481 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" event={"ID":"6d788ee1-cd0c-486c-9ee4-3bd725f1433b","Type":"ContainerDied","Data":"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649"} Oct 30 20:27:02.077993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.077543 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4" event={"ID":"6d788ee1-cd0c-486c-9ee4-3bd725f1433b","Type":"ContainerDied","Data":"f3ce7361026a9858b43f6f26f7146d1c491bcc000ca974cd18a4336bc7eb7f7b"} Oct 30 20:27:02.077993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.077571 4524 scope.go:122] "RemoveContainer" containerID="77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649" Oct 30 20:27:02.088152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.088102 4524 scope.go:122] "RemoveContainer" containerID="77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649" Oct 30 20:27:02.088394 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:02.088374 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649\": container with ID starting with 77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649 not found: ID does not exist" containerID="77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649" Oct 30 20:27:02.088434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.088404 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649"} err="failed to get container status \"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649\": rpc error: code = NotFound desc = could not find container \"77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649\": container with ID starting with 77aca7e85dc71fa502afbb0dacbf569c73095d67fbca6dc78dd5a348f6155649 not found: ID does not exist" Oct 30 20:27:02.243263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.243231 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:27:02.293004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:02.292977 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0051fb-67bb67b7f9-9npc4"] Oct 30 20:27:03.038935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:03.038905 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:03.299999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:03.299897 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" path="/var/lib/kubelet/pods/6d788ee1-cd0c-486c-9ee4-3bd725f1433b/volumes" Oct 30 20:27:05.530655 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:05.530621 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:05.699037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:05.698996 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerName="payload-processing" containerID="cri-o://93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104" gracePeriod=25 Oct 30 20:27:05.724600 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:27:05.724579 4524 logging.go:55] [core] [Channel #1577 SubChannel #1578]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.126:50052", ServerName: "10.133.0.126:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.126:50052: connect: connection refused" Oct 30 20:27:05.809382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:05.809285 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" containerID="cri-o://3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb" gracePeriod=25 Oct 30 20:27:05.953167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:05.953132 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:27:06.079949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.079273 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\") pod \"4bd70019-8732-45c7-a432-9d78df73b7ba\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " Oct 30 20:27:06.079949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.079341 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\" (UniqueName: \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\") pod \"4bd70019-8732-45c7-a432-9d78df73b7ba\" (UID: \"4bd70019-8732-45c7-a432-9d78df73b7ba\") " Oct 30 20:27:06.080169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.080030 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume" pod "4bd70019-8732-45c7-a432-9d78df73b7ba" (UID: "4bd70019-8732-45c7-a432-9d78df73b7ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:06.082733 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.082257 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp" pod "4bd70019-8732-45c7-a432-9d78df73b7ba" (UID: "4bd70019-8732-45c7-a432-9d78df73b7ba"). InnerVolumeSpecName "kube-api-access-t95qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:06.101882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.101854 4524 generic.go:356] "Generic (PLEG): container finished" podID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerID="93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104" exitCode=0 Oct 30 20:27:06.102001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.101888 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" event={"ID":"4bd70019-8732-45c7-a432-9d78df73b7ba","Type":"ContainerDied","Data":"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104"} Oct 30 20:27:06.102001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.101919 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" Oct 30 20:27:06.102001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.101953 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" event={"ID":"4bd70019-8732-45c7-a432-9d78df73b7ba","Type":"ContainerDied","Data":"6b0d85889e6b6991bc6e791fc8548fba6482a83821026e302f27791fc989c8f1"} Oct 30 20:27:06.102001 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.101977 4524 scope.go:122] "RemoveContainer" containerID="93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104" Oct 30 20:27:06.110679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.110662 4524 scope.go:122] "RemoveContainer" containerID="93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104" Oct 30 20:27:06.111165 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:06.111143 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104\": container with ID starting with 93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104 not found: ID does not exist" containerID="93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104" Oct 30 20:27:06.111247 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.111172 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104"} err="failed to get container status \"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104\": rpc error: code = NotFound desc = could not find container \"93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104\": container with ID starting with 93f0b280b77c41c84643118b0b5f12fc0e944407d8a99c6043f74fc05c01d104 not found: ID does not exist" Oct 30 20:27:06.181238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.181205 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd70019-8732-45c7-a432-9d78df73b7ba-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:06.181238 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.181240 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95qp\" (UniqueName: \"kubernetes.io/projected/4bd70019-8732-45c7-a432-9d78df73b7ba-kube-api-access-t95qp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:06.267319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.267285 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:27:06.272461 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.272432 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf"] Oct 30 20:27:06.724853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:06.724787 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0051fb-dd848dc88-p6dxf" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.126:50052\" within 1s: context deadline exceeded" Oct 30 20:27:07.300551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:07.300518 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" path="/var/lib/kubelet/pods/4bd70019-8732-45c7-a432-9d78df73b7ba/volumes" Oct 30 20:27:08.873675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:08.873640 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:12.786353 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:12.786309 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:27:13.157897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:13.157863 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:13.387398 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:13.387359 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:13.632343 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:13.632269 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:27:13.632620 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:13.632449 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" containerID="cri-o://4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9" gracePeriod=30 Oct 30 20:27:13.705755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:13.705557 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:14.694800 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:14.694758 4524 patch_prober.go:29] interesting pod/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:27:14.695198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:14.694817 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:27:14.719793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:14.719755 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:27:18.289193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:18.289154 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.901291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:18.901232 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:27:19.025949 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.025920 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.025961 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026004 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026173 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026212 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026238 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026272 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:19.026425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026302 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026362 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026358 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:19.026425 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026395 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\") pod \"8286b115-c193-457f-8882-bf43b1fe4b3c\" (UID: \"8286b115-c193-457f-8882-bf43b1fe4b3c\") " Oct 30 20:27:19.026693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026548 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:19.026693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026619 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:19.026693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026655 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:19.026871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026851 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.026940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026877 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8286b115-c193-457f-8882-bf43b1fe4b3c-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.026940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026890 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.026940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026898 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.026940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.026907 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.028344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.028320 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:19.028725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.028679 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:19.028725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.028686 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:27:19.028842 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.028729 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2" pod "8286b115-c193-457f-8882-bf43b1fe4b3c" (UID: "8286b115-c193-457f-8882-bf43b1fe4b3c"). InnerVolumeSpecName "kube-api-access-ghlb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:19.128261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.128217 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghlb2\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-kube-api-access-ghlb2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.128261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.128257 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.128261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.128267 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.128261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.128276 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8286b115-c193-457f-8882-bf43b1fe4b3c-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:19.191348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.191263 4524 generic.go:356] "Generic (PLEG): container finished" podID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerID="4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9" exitCode=0 Oct 30 20:27:19.191348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.191313 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerDied","Data":"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9"} Oct 30 20:27:19.191348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.191323 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" Oct 30 20:27:19.191597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.191367 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6" event={"ID":"8286b115-c193-457f-8882-bf43b1fe4b3c","Type":"ContainerDied","Data":"643554991b142fdc9e0f9fe9e7b2799e151371e0925c2c9beecf9e20a85c9186"} Oct 30 20:27:19.191597 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.191399 4524 scope.go:122] "RemoveContainer" containerID="4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9" Oct 30 20:27:19.205097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.205076 4524 scope.go:122] "RemoveContainer" containerID="5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5" Oct 30 20:27:19.213868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.213849 4524 scope.go:122] "RemoveContainer" containerID="4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9" Oct 30 20:27:19.214123 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:19.214101 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9\": container with ID starting with 4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9 not found: ID does not exist" containerID="4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9" Oct 30 20:27:19.214179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.214140 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9"} err="failed to get container status \"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9\": rpc error: code = NotFound desc = could not find container \"4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9\": container with ID starting with 4d90830c87210e70f8bc0237dcf5b3d17ab7f3b9db30f9f73504fc6c3a1d24d9 not found: ID does not exist" Oct 30 20:27:19.214179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.214162 4524 scope.go:122] "RemoveContainer" containerID="5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5" Oct 30 20:27:19.214402 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:19.214381 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5\": container with ID starting with 5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5 not found: ID does not exist" containerID="5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5" Oct 30 20:27:19.214443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.214409 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5"} err="failed to get container status \"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5\": rpc error: code = NotFound desc = could not find container \"5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5\": container with ID starting with 5b27eca1530f65313c35d5dcfe24a2f8793e82c2220245214a90d356d64051e5 not found: ID does not exist" Oct 30 20:27:19.357942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.357910 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:27:19.361412 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:19.361390 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0051fb-openshift-default-567bd7fc87-kmqk6"] Oct 30 20:27:21.299863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:21.299828 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" path="/var/lib/kubelet/pods/8286b115-c193-457f-8882-bf43b1fe4b3c/volumes" Oct 30 20:27:23.103931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:23.103898 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:24.720057 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:24.720009 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:27:26.070072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.070019 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:27:26.088937 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.088916 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:27:26.094939 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.094921 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\") pod \"b3870242-9d90-4662-8a01-d06fe5defbc1\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " Oct 30 20:27:26.095000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.094951 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\" (UniqueName: \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\") pod \"b3870242-9d90-4662-8a01-d06fe5defbc1\" (UID: \"b3870242-9d90-4662-8a01-d06fe5defbc1\") " Oct 30 20:27:26.095265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.095245 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume" pod "b3870242-9d90-4662-8a01-d06fe5defbc1" (UID: "b3870242-9d90-4662-8a01-d06fe5defbc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:26.097063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.097038 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv" pod "b3870242-9d90-4662-8a01-d06fe5defbc1" (UID: "b3870242-9d90-4662-8a01-d06fe5defbc1"). InnerVolumeSpecName "kube-api-access-tlrnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:26.196538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.196506 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3870242-9d90-4662-8a01-d06fe5defbc1-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:26.196728 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.196560 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrnv\" (UniqueName: \"kubernetes.io/projected/b3870242-9d90-4662-8a01-d06fe5defbc1-kube-api-access-tlrnv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:26.232065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.232038 4524 generic.go:356] "Generic (PLEG): container finished" podID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerID="3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb" exitCode=0 Oct 30 20:27:26.232193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.232082 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" Oct 30 20:27:26.232193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.232114 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" event={"ID":"b3870242-9d90-4662-8a01-d06fe5defbc1","Type":"ContainerDied","Data":"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb"} Oct 30 20:27:26.232193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.232153 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v" event={"ID":"b3870242-9d90-4662-8a01-d06fe5defbc1","Type":"ContainerDied","Data":"a0b66a0df58ed7457011b38214f4db5b0bc8112d20f245dbae2ed3c7449822a4"} Oct 30 20:27:26.232193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.232174 4524 scope.go:122] "RemoveContainer" containerID="3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb" Oct 30 20:27:26.242149 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.242122 4524 scope.go:122] "RemoveContainer" containerID="3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb" Oct 30 20:27:26.242426 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:26.242408 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb\": container with ID starting with 3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb not found: ID does not exist" containerID="3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb" Oct 30 20:27:26.242497 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.242432 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb"} err="failed to get container status \"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb\": rpc error: code = NotFound desc = could not find container \"3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb\": container with ID starting with 3797af105b4e5aea032c5948976a90a5cbd374c991acff133bfaa2fca0b4fbfb not found: ID does not exist" Oct 30 20:27:26.407307 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.407276 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:26.419211 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.419184 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:27:26.424300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.424276 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0051fb-77f75c56b7-vll5v"] Oct 30 20:27:26.636860 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:26.636826 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:27.299594 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:27.299564 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" path="/var/lib/kubelet/pods/b3870242-9d90-4662-8a01-d06fe5defbc1/volumes" Oct 30 20:27:27.878871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:27.873140 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:31.071266 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:31.071206 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" containerID="cri-o://88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba" gracePeriod=25 Oct 30 20:27:32.706721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:32.704103 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:33.229542 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:33.229503 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:27:37.647142 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:37.647108 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:42.417609 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:42.417570 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:42.785997 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:42.785911 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:27:43.229511 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:43.229469 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:27:47.335570 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:47.335518 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:48.942405 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:48.942376 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:50.181053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:50.181020 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:51.332264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.332205 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:27:51.374491 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.374461 4524 generic.go:356] "Generic (PLEG): container finished" podID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerID="88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba" exitCode=0 Oct 30 20:27:51.374621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.374509 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" Oct 30 20:27:51.374621 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.374551 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" event={"ID":"d3b6ca18-647d-4394-a749-10fba8eb4be3","Type":"ContainerDied","Data":"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba"} Oct 30 20:27:51.374732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.374622 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw" event={"ID":"d3b6ca18-647d-4394-a749-10fba8eb4be3","Type":"ContainerDied","Data":"6751d3e715438b738385a02fc70dfdf5dc63eda75451715865a36f7d35ad49e5"} Oct 30 20:27:51.374732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.374654 4524 scope.go:122] "RemoveContainer" containerID="88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba" Oct 30 20:27:51.384874 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.384852 4524 scope.go:122] "RemoveContainer" containerID="88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba" Oct 30 20:27:51.385212 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:51.385190 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba\": container with ID starting with 88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba not found: ID does not exist" containerID="88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba" Oct 30 20:27:51.385324 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.385218 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba"} err="failed to get container status \"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba\": rpc error: code = NotFound desc = could not find container \"88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba\": container with ID starting with 88fead2a8cd5e049cdf57bc67eeadf4986ac38007fd0e06acaa03c4dd6d502ba not found: ID does not exist" Oct 30 20:27:51.386724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.386681 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:27:51.386899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.386882 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" podUID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" containerName="maas-api" containerID="cri-o://86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056" gracePeriod=30 Oct 30 20:27:51.446885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.446856 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\") pod \"d3b6ca18-647d-4394-a749-10fba8eb4be3\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " Oct 30 20:27:51.447065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.447006 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\" (UniqueName: \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\") pod \"d3b6ca18-647d-4394-a749-10fba8eb4be3\" (UID: \"d3b6ca18-647d-4394-a749-10fba8eb4be3\") " Oct 30 20:27:51.447292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.447267 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume" pod "d3b6ca18-647d-4394-a749-10fba8eb4be3" (UID: "d3b6ca18-647d-4394-a749-10fba8eb4be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:51.447436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.447420 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b6ca18-647d-4394-a749-10fba8eb4be3-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.449470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.449447 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt" pod "d3b6ca18-647d-4394-a749-10fba8eb4be3" (UID: "d3b6ca18-647d-4394-a749-10fba8eb4be3"). InnerVolumeSpecName "kube-api-access-x8rjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:51.544989 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.544952 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:51.548318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.548290 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rjt\" (UniqueName: \"kubernetes.io/projected/d3b6ca18-647d-4394-a749-10fba8eb4be3-kube-api-access-x8rjt\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.647187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.647159 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:27:51.750740 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.750683 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") pod \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " Oct 30 20:27:51.750875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.750785 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\") pod \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " Oct 30 20:27:51.750875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.750809 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\") pod \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " Oct 30 20:27:51.751020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.750924 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\" (UniqueName: \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\") pod \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\" (UID: \"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8\") " Oct 30 20:27:51.751194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.751171 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca" pod "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" (UID: "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:51.752840 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.752816 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls" pod "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" (UID: "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:51.752935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.752906 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x" pod "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" (UID: "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8"). InnerVolumeSpecName "kube-api-access-w998x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:51.752982 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.752948 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs" pod "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" (UID: "28a9f0ac-f3aa-417a-9545-0ed992a1bfa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:51.851942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.851909 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w998x\" (UniqueName: \"kubernetes.io/projected/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-kube-api-access-w998x\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.851942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.851935 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.851942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.851947 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.852186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.851955 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:51.922203 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.922114 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:27:51.925164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:51.925127 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2386c-6cc9f9f46c-4vqpw"] Oct 30 20:27:52.381884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.381798 4524 generic.go:356] "Generic (PLEG): container finished" podID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" containerID="86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056" exitCode=0 Oct 30 20:27:52.381884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.381851 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" Oct 30 20:27:52.382447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.381876 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" event={"ID":"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8","Type":"ContainerDied","Data":"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056"} Oct 30 20:27:52.382447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.381933 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r" event={"ID":"28a9f0ac-f3aa-417a-9545-0ed992a1bfa8","Type":"ContainerDied","Data":"f50b2e74db6ce5834385d678030a9a5e61bb115711175c315620fbe38b804bde"} Oct 30 20:27:52.382447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.381955 4524 scope.go:122] "RemoveContainer" containerID="86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056" Oct 30 20:27:52.393166 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.393148 4524 scope.go:122] "RemoveContainer" containerID="86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056" Oct 30 20:27:52.393436 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:27:52.393412 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056\": container with ID starting with 86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056 not found: ID does not exist" containerID="86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056" Oct 30 20:27:52.393568 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.393442 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056"} err="failed to get container status \"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056\": rpc error: code = NotFound desc = could not find container \"86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056\": container with ID starting with 86e20ee10a0cdd1faa8085395555f2bd5afe4d3de0f584a3722f99dcd8ed8056 not found: ID does not exist" Oct 30 20:27:52.548301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.548263 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:27:52.597234 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:52.597199 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2386c-9dc8f4cfc-fb48r"] Oct 30 20:27:53.112156 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:53.112125 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.298940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:53.298904 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" path="/var/lib/kubelet/pods/28a9f0ac-f3aa-417a-9545-0ed992a1bfa8/volumes" Oct 30 20:27:53.299292 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:53.299277 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" path="/var/lib/kubelet/pods/d3b6ca18-647d-4394-a749-10fba8eb4be3/volumes" Oct 30 20:27:53.397885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:53.397850 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.621463 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:53.621428 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:54.058515 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:54.058482 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:58.366675 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:58.366641 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:59.182973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:59.182937 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:59.397299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:27:59.397270 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.074525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:01.074491 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.327188 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:01.327090 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.801187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:01.801134 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:02.056898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:02.056816 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:03.462824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:03.462788 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:06.900209 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:06.900172 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:28:07.093611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:07.093574 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.558954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:08.558913 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:12.786267 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:28:12.786232 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:28:13.349996 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:13.349965 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:14.357307 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:14.357272 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:14.655617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:14.655587 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:18.305164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:18.305125 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:21.920096 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:21.920053 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" containerID="cri-o://e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae" gracePeriod=45 Oct 30 20:28:22.178032 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.177973 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:28:22.278434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278399 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278466 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278515 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\" (UniqueName: \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278593 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278572 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278627 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278638 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:22.278788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278651 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\") pod \"3896eeb3-6acb-424e-954b-9d5e35634d7b\" (UID: \"3896eeb3-6acb-424e-954b-9d5e35634d7b\") " Oct 30 20:28:22.278788 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278776 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:22.279018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.278899 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:22.279121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.279102 4524 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-home\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.279159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.279126 4524 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-tmp-dir\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.279159 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.279140 4524 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-model-cache\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.280935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.280908 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:22.280935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.280924 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "kube-api-access-55h9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:22.281086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.280946 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm" pod "3896eeb3-6acb-424e-954b-9d5e35634d7b" (UID: "3896eeb3-6acb-424e-954b-9d5e35634d7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:22.382044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.380873 4524 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3896eeb3-6acb-424e-954b-9d5e35634d7b-dshm\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.382044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.380911 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55h9p\" (UniqueName: \"kubernetes.io/projected/3896eeb3-6acb-424e-954b-9d5e35634d7b-kube-api-access-55h9p\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.382044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.380927 4524 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3896eeb3-6acb-424e-954b-9d5e35634d7b-tls-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:22.568861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.568777 4524 generic.go:356] "Generic (PLEG): container finished" podID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerID="e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae" exitCode=0 Oct 30 20:28:22.568861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.568829 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" Oct 30 20:28:22.568861 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.568850 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" event={"ID":"3896eeb3-6acb-424e-954b-9d5e35634d7b","Type":"ContainerDied","Data":"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae"} Oct 30 20:28:22.569128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.568880 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx" event={"ID":"3896eeb3-6acb-424e-954b-9d5e35634d7b","Type":"ContainerDied","Data":"0f6221e2bbc3179c0de836803500492c57ba78880833017c2de316b9c010bed1"} Oct 30 20:28:22.569128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.568896 4524 scope.go:122] "RemoveContainer" containerID="e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae" Oct 30 20:28:22.579262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.579243 4524 scope.go:122] "RemoveContainer" containerID="e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae" Oct 30 20:28:22.579539 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:28:22.579518 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae\": container with ID starting with e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae not found: ID does not exist" containerID="e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae" Oct 30 20:28:22.579606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.579552 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae"} err="failed to get container status \"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae\": rpc error: code = NotFound desc = could not find container \"e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae\": container with ID starting with e8ceff6aa8cf412b638f2c3ece6204b0665f825f2db6ca9da30138f4a8c87dae not found: ID does not exist" Oct 30 20:28:22.740015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.739976 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:28:22.789181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:22.789146 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779921/premium-w0-779921-kserve-9f646dd9-gzrfx"] Oct 30 20:28:23.136606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:23.136574 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:23.299148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:23.299118 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" path="/var/lib/kubelet/pods/3896eeb3-6acb-424e-954b-9d5e35634d7b/volumes" Oct 30 20:28:27.974084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:27.974047 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:32.458380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:32.458340 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:32.862582 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:32.862499 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.138317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:38.138282 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:42.784580 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:28:42.784541 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:28:43.180751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:43.177500 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:43.397140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:43.397105 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:45.952572 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:45.952535 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:46.173302 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:46.173266 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:46.424099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:46.424066 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:47.196538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.196503 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:28:47.196948 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.196752 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" podUID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" containerName="maas-api" containerID="cri-o://283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672" gracePeriod=30 Oct 30 20:28:47.467462 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.467437 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:28:47.542164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.542119 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:47.561229 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561202 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\") pod \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " Oct 30 20:28:47.561389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561262 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") pod \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " Oct 30 20:28:47.561389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561338 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\") pod \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " Oct 30 20:28:47.561389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561366 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\" (UniqueName: \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\") pod \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\" (UID: \"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25\") " Oct 30 20:28:47.561606 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561527 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca" pod "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" (UID: "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:47.561775 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.561749 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:47.563576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.563547 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls" pod "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" (UID: "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:47.563669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.563588 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs" pod "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" (UID: "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:47.563669 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.563638 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2" pod "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" (UID: "c0c8ea09-5ca6-412f-bda8-63a5e68a1a25"). InnerVolumeSpecName "kube-api-access-h6mb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:47.662720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.662678 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:47.662875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.662730 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:47.662875 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.662741 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6mb2\" (UniqueName: \"kubernetes.io/projected/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25-kube-api-access-h6mb2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:47.700423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.700391 4524 generic.go:356] "Generic (PLEG): container finished" podID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" containerID="283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672" exitCode=0 Oct 30 20:28:47.700570 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.700437 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" Oct 30 20:28:47.700570 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.700482 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" event={"ID":"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25","Type":"ContainerDied","Data":"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672"} Oct 30 20:28:47.700570 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.700551 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s" event={"ID":"c0c8ea09-5ca6-412f-bda8-63a5e68a1a25","Type":"ContainerDied","Data":"d9d9cba52bc07929ba831fb08232eb003d0ead06f8e83f64cd1d511a86043a82"} Oct 30 20:28:47.700714 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.700578 4524 scope.go:122] "RemoveContainer" containerID="283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672" Oct 30 20:28:47.710375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.710359 4524 scope.go:122] "RemoveContainer" containerID="283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672" Oct 30 20:28:47.710631 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:28:47.710610 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672\": container with ID starting with 283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672 not found: ID does not exist" containerID="283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672" Oct 30 20:28:47.710721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.710645 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672"} err="failed to get container status \"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672\": rpc error: code = NotFound desc = could not find container \"283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672\": container with ID starting with 283ed32cffe5ee444a6fc329253ffaf7b6b64de8463755b0039a88bbdde20672 not found: ID does not exist" Oct 30 20:28:47.865724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.865632 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:28:47.868872 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:47.868847 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-01e8dc-59649f5c7d-8wp7s"] Oct 30 20:28:49.042269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:49.042238 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:49.299063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:49.298989 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" path="/var/lib/kubelet/pods/c0c8ea09-5ca6-412f-bda8-63a5e68a1a25/volumes" Oct 30 20:28:49.327933 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:49.327896 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:49.581681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:49.581581 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:49.995960 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:49.995928 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:50.413625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:50.413590 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:50.621872 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:50.621836 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.107305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:53.107271 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:54.591834 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:54.591805 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:58.264223 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:58.264141 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:58.630931 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:58.630851 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:58.673198 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:58.673167 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:28:58.673453 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:58.673424 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" containerID="cri-o://4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8" gracePeriod=30 Oct 30 20:28:58.970217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:58.970184 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:59.380546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:28:59.380438 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:03.931570 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:03.931543 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:29:04.034313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034225 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034277 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034345 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034375 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034507 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034550 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034591 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034607 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:04.034839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034741 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.034839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034778 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\") pod \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\" (UID: \"d5238180-2ad7-4b20-bec7-5c244c68c2d0\") " Oct 30 20:29:04.035064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.034947 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:04.035064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035014 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:04.035177 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035107 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:04.035177 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035147 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:04.035332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035308 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.035332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035334 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.035552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035350 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.035552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035367 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.035552 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.035379 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.036671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.036642 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:04.036671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.036656 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:29:04.036671 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.036649 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "kube-api-access-gsqc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:04.036902 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.036841 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token" pod "d5238180-2ad7-4b20-bec7-5c244c68c2d0" (UID: "d5238180-2ad7-4b20-bec7-5c244c68c2d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:04.136321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.136286 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.136321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.136317 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqc5\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-kube-api-access-gsqc5\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.136513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.136332 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.136513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.136344 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5238180-2ad7-4b20-bec7-5c244c68c2d0-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:04.680538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.680493 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:04.785035 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.785003 4524 generic.go:356] "Generic (PLEG): container finished" podID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerID="4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8" exitCode=0 Oct 30 20:29:04.785195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.785059 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerDied","Data":"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8"} Oct 30 20:29:04.785195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.785069 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" Oct 30 20:29:04.785195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.785146 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t" event={"ID":"d5238180-2ad7-4b20-bec7-5c244c68c2d0","Type":"ContainerDied","Data":"c71df95b473605468d4f0d2bb847620937d76b2a5148b4b186c25e3265d1e339"} Oct 30 20:29:04.785195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.785177 4524 scope.go:122] "RemoveContainer" containerID="4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8" Oct 30 20:29:04.795480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.795319 4524 scope.go:122] "RemoveContainer" containerID="d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554" Oct 30 20:29:04.803963 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.803937 4524 scope.go:122] "RemoveContainer" containerID="4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8" Oct 30 20:29:04.804274 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:04.804252 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8\": container with ID starting with 4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8 not found: ID does not exist" containerID="4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8" Oct 30 20:29:04.804355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.804285 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8"} err="failed to get container status \"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8\": rpc error: code = NotFound desc = could not find container \"4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8\": container with ID starting with 4336e443e0a2a1a5ee7956e5b64d8f1e03e2eeca253e775b0db514507a660bc8 not found: ID does not exist" Oct 30 20:29:04.804355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.804304 4524 scope.go:122] "RemoveContainer" containerID="d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554" Oct 30 20:29:04.804572 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:04.804555 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554\": container with ID starting with d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554 not found: ID does not exist" containerID="d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554" Oct 30 20:29:04.804631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:04.804578 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554"} err="failed to get container status \"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554\": rpc error: code = NotFound desc = could not find container \"d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554\": container with ID starting with d047c2d6c552256ede278d389aff9ddb6248001f6eeb41b72ee4895c52d7b554 not found: ID does not exist" Oct 30 20:29:05.003966 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:05.003885 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:29:05.053882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:05.053857 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-01e8dc-openshift-default-588b59fc77-vlc5t"] Oct 30 20:29:05.299323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:05.299250 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" path="/var/lib/kubelet/pods/d5238180-2ad7-4b20-bec7-5c244c68c2d0/volumes" Oct 30 20:29:09.479389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:09.479360 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:09.696853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:09.696821 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.414980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:11.414946 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.628993 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:11.628956 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:12.328010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:12.327974 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:29:12.389521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:12.389490 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:12.787309 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:12.787271 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:29:14.468025 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:14.467992 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:15.932005 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:15.931940 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:16.146987 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:16.146936 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:16.982003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:16.981974 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:17.219591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:17.219556 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:17.449576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:17.449542 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.606540 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.606505 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 20:29:18.607012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.606773 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" podUID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" containerName="maas-api" containerID="cri-o://875ba5e0a303050788b7fc86a6eef1f7b475d81ca64c6ada518bb9731bfd9fa2" gracePeriod=30 Oct 30 20:29:18.768771 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.768727 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.861264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.861193 4524 generic.go:356] "Generic (PLEG): container finished" podID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" containerID="875ba5e0a303050788b7fc86a6eef1f7b475d81ca64c6ada518bb9731bfd9fa2" exitCode=0 Oct 30 20:29:18.861378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.861261 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" event={"ID":"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212","Type":"ContainerDied","Data":"875ba5e0a303050788b7fc86a6eef1f7b475d81ca64c6ada518bb9731bfd9fa2"} Oct 30 20:29:18.861378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.861321 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" event={"ID":"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212","Type":"ContainerDied","Data":"0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3"} Oct 30 20:29:18.861378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.861340 4524 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a28bbf92750b8b9ca81360a6a4ee77418195569016a96a69735557b4f7621d3" Oct 30 20:29:18.869853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.869831 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 20:29:18.887419 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.887396 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\") pod \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " Oct 30 20:29:18.887548 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.887497 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\" (UniqueName: \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\") pod \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " Oct 30 20:29:18.887548 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.887538 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") pod \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " Oct 30 20:29:18.887628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.887612 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\") pod \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\" (UID: \"fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212\") " Oct 30 20:29:18.888041 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.888015 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca" pod "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" (UID: "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:18.889678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.889648 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb" pod "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" (UID: "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212"). InnerVolumeSpecName "kube-api-access-jvwtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:18.889887 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.889869 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs" pod "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" (UID: "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:18.889960 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.889888 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls" pod "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" (UID: "fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:18.992615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.992539 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:18.992615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.992573 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:18.992615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.992583 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwtb\" (UniqueName: \"kubernetes.io/projected/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-kube-api-access-jvwtb\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:18.992615 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:18.992593 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:19.868285 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:19.868250 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq" Oct 30 20:29:20.034184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:20.034153 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 20:29:20.084827 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:20.084793 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-779921-98c756c89-k9xlq"] Oct 30 20:29:20.329286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:20.329251 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:20.559323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:20.559291 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:20.791366 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:20.791330 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:21.296889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:21.296858 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:21.298893 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:21.298873 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" path="/var/lib/kubelet/pods/fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212/volumes" Oct 30 20:29:25.088244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:25.088196 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:26.625883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:26.625846 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:26.869028 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:26.868994 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:27.128754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.128716 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:27.347958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.347923 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" podUID="bce877b5-0218-4c55-8d5d-25083e1894ea" containerName="main" containerID="cri-o://dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba" gracePeriod=45 Oct 30 20:29:27.553365 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.553298 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:27.604812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.604786 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:29:27.680483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680453 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680511 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680577 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680605 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680682 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\" (UniqueName: \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680758 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\") pod \"bce877b5-0218-4c55-8d5d-25083e1894ea\" (UID: \"bce877b5-0218-4c55-8d5d-25083e1894ea\") " Oct 30 20:29:27.680943 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.680858 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:27.681244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.681093 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:27.681244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.681106 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:27.681244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.681218 4524 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-tmp-dir\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.681244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.681239 4524 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-model-cache\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.681382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.681254 4524 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-home\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.683349 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.683324 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:27.683488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.683436 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "kube-api-access-dfzgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:27.684791 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.684769 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs" pod "bce877b5-0218-4c55-8d5d-25083e1894ea" (UID: "bce877b5-0218-4c55-8d5d-25083e1894ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:27.782135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.782039 4524 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce877b5-0218-4c55-8d5d-25083e1894ea-tls-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.782135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.782090 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzgp\" (UniqueName: \"kubernetes.io/projected/bce877b5-0218-4c55-8d5d-25083e1894ea-kube-api-access-dfzgp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.782135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.782107 4524 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce877b5-0218-4c55-8d5d-25083e1894ea-dshm\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:27.911693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.911658 4524 generic.go:356] "Generic (PLEG): container finished" podID="bce877b5-0218-4c55-8d5d-25083e1894ea" containerID="dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba" exitCode=0 Oct 30 20:29:27.911905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.911725 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" Oct 30 20:29:27.911905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.911737 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" event={"ID":"bce877b5-0218-4c55-8d5d-25083e1894ea","Type":"ContainerDied","Data":"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba"} Oct 30 20:29:27.911905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.911778 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd" event={"ID":"bce877b5-0218-4c55-8d5d-25083e1894ea","Type":"ContainerDied","Data":"16d85153b46c7afecfe84f687f3f3702341b2e74ca11721452015c477da8d91a"} Oct 30 20:29:27.911905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.911798 4524 scope.go:122] "RemoveContainer" containerID="dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba" Oct 30 20:29:27.921493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.921478 4524 scope.go:122] "RemoveContainer" containerID="dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba" Oct 30 20:29:27.921772 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:27.921754 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba\": container with ID starting with dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba not found: ID does not exist" containerID="dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba" Oct 30 20:29:27.921841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:27.921782 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba"} err="failed to get container status \"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba\": rpc error: code = NotFound desc = could not find container \"dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba\": container with ID starting with dcddc033b556b914ad973da06c0c190dfabb439b33e86a587afd59af3b6202ba not found: ID does not exist" Oct 30 20:29:28.051283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:28.051194 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:28.084338 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:28.084308 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:29:28.087627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:28.087600 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-14c846/test-model-14c846-kserve-6db6fdb5b7-k89rd"] Oct 30 20:29:28.284848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:28.284802 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:28.553086 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:28.553048 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:29.298831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:29.298795 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce877b5-0218-4c55-8d5d-25083e1894ea" path="/var/lib/kubelet/pods/bce877b5-0218-4c55-8d5d-25083e1894ea/volumes" Oct 30 20:29:31.998952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:31.998917 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:34.627011 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:34.626969 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:34.864870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:34.864834 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:34.936921 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:34.936891 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:29:34.937148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:34.937126 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" containerID="cri-o://c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a" gracePeriod=30 Oct 30 20:29:35.101313 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:35.101284 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:35.484648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:35.484610 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:37.041241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:37.041191 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:40.195814 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.195790 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:29:40.307499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307429 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307482 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307511 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307542 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307633 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307686 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307675 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307725 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307788 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307855 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\") pod \"463488c8-7f01-4773-a3c8-72dc66418507\" (UID: \"463488c8-7f01-4773-a3c8-72dc66418507\") " Oct 30 20:29:40.307926 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307877 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:40.308161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.307966 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:40.308161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308048 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:40.308161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308134 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:40.308332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308231 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:40.308399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308353 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.308399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308373 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.308399 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308387 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.308554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308402 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463488c8-7f01-4773-a3c8-72dc66418507-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.308554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.308418 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.310040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.310009 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:40.310199 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.310056 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:29:40.310344 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.310321 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "kube-api-access-sjfnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:40.310494 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.310379 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token" pod "463488c8-7f01-4773-a3c8-72dc66418507" (UID: "463488c8-7f01-4773-a3c8-72dc66418507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:40.410241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.410207 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463488c8-7f01-4773-a3c8-72dc66418507-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.410241 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.410240 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.410452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.410253 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463488c8-7f01-4773-a3c8-72dc66418507-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.410452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.410272 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjfnp\" (UniqueName: \"kubernetes.io/projected/463488c8-7f01-4773-a3c8-72dc66418507-kube-api-access-sjfnp\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:40.419746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.419715 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:29:40.613812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.613720 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:40.979242 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.979204 4524 generic.go:356] "Generic (PLEG): container finished" podID="463488c8-7f01-4773-a3c8-72dc66418507" containerID="c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a" exitCode=0 Oct 30 20:29:40.979411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.979274 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" Oct 30 20:29:40.979411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.979291 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerDied","Data":"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a"} Oct 30 20:29:40.979411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.979392 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd" event={"ID":"463488c8-7f01-4773-a3c8-72dc66418507","Type":"ContainerDied","Data":"8383803b0438edf93b20cd870bc9aa2b41782635d4ab6537378bed013cc678ee"} Oct 30 20:29:40.979587 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.979422 4524 scope.go:122] "RemoveContainer" containerID="c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a" Oct 30 20:29:40.989454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.989434 4524 scope.go:122] "RemoveContainer" containerID="9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf" Oct 30 20:29:40.997591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.997569 4524 scope.go:122] "RemoveContainer" containerID="c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a" Oct 30 20:29:40.997912 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:40.997892 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a\": container with ID starting with c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a not found: ID does not exist" containerID="c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a" Oct 30 20:29:40.998014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.997926 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a"} err="failed to get container status \"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a\": rpc error: code = NotFound desc = could not find container \"c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a\": container with ID starting with c16e20df600db4ee38ab5cd8933396d1a6279a7278405b5ecb75da0d8c75d24a not found: ID does not exist" Oct 30 20:29:40.998014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.997951 4524 scope.go:122] "RemoveContainer" containerID="9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf" Oct 30 20:29:40.998219 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:40.998197 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf\": container with ID starting with 9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf not found: ID does not exist" containerID="9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf" Oct 30 20:29:40.998263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:40.998226 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf"} err="failed to get container status \"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf\": rpc error: code = NotFound desc = could not find container \"9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf\": container with ID starting with 9afb89eede4d7719d40d9362e65668061682f402e5b6c67f778f4a16ee7e8cbf not found: ID does not exist" Oct 30 20:29:41.145215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:41.145181 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:29:41.195372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:41.195342 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-453277-openshift-default-5b497cd777-bbwqd"] Oct 30 20:29:41.299384 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:41.299310 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463488c8-7f01-4773-a3c8-72dc66418507" path="/var/lib/kubelet/pods/463488c8-7f01-4773-a3c8-72dc66418507/volumes" Oct 30 20:29:41.927396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:41.927357 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:42.784649 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:42.784614 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:29:46.835099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:46.835067 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:47.411556 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:47.411522 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:47.485311 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:47.485281 4524 scope.go:122] "RemoveContainer" containerID="875ba5e0a303050788b7fc86a6eef1f7b475d81ca64c6ada518bb9731bfd9fa2" Oct 30 20:29:50.925642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:50.925607 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:51.996350 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:51.996304 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:52.242195 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:52.242135 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:53.227732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:53.227673 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:55.441518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.441484 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" containerID="cri-o://1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b" gracePeriod=45 Oct 30 20:29:55.688961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.688929 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:29:55.757679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757608 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.757838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757686 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\" (UniqueName: \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.757838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757732 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\" (UniqueName: \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.757838 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757780 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.758016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757834 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:55.758016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757846 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.758016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.757910 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\") pod \"6262a9d5-36ff-4999-b1d6-1191b8303485\" (UID: \"6262a9d5-36ff-4999-b1d6-1191b8303485\") " Oct 30 20:29:55.758167 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.758064 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:55.758219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.758164 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:55.758286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.758271 4524 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-home\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:55.758342 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.758288 4524 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-tmp-dir\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:55.758342 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.758299 4524 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-model-cache\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:55.759962 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.759940 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:55.760374 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.760351 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:55.760443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.760372 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2" pod "6262a9d5-36ff-4999-b1d6-1191b8303485" (UID: "6262a9d5-36ff-4999-b1d6-1191b8303485"). InnerVolumeSpecName "kube-api-access-8bsj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:55.859116 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.859083 4524 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6262a9d5-36ff-4999-b1d6-1191b8303485-dshm\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:55.859116 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.859114 4524 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6262a9d5-36ff-4999-b1d6-1191b8303485-tls-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:55.859355 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:55.859127 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bsj2\" (UniqueName: \"kubernetes.io/projected/6262a9d5-36ff-4999-b1d6-1191b8303485-kube-api-access-8bsj2\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:29:56.055018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.054925 4524 generic.go:356] "Generic (PLEG): container finished" podID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerID="1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b" exitCode=0 Oct 30 20:29:56.055018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.054987 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" Oct 30 20:29:56.055221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.055005 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" event={"ID":"6262a9d5-36ff-4999-b1d6-1191b8303485","Type":"ContainerDied","Data":"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b"} Oct 30 20:29:56.055221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.055060 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx" event={"ID":"6262a9d5-36ff-4999-b1d6-1191b8303485","Type":"ContainerDied","Data":"b56572ac2aff94ee533053895a803de20f6f6ed0a4f8424fa13a72f57e90e369"} Oct 30 20:29:56.055221 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.055077 4524 scope.go:122] "RemoveContainer" containerID="1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b" Oct 30 20:29:56.063819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.063804 4524 scope.go:122] "RemoveContainer" containerID="1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b" Oct 30 20:29:56.064119 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:29:56.064100 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b\": container with ID starting with 1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b not found: ID does not exist" containerID="1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b" Oct 30 20:29:56.064169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.064128 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b"} err="failed to get container status \"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b\": rpc error: code = NotFound desc = could not find container \"1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b\": container with ID starting with 1d4265ce100785a04a3c41894e4c771cb8e1e02722626ae2b981d0e0f247960b not found: ID does not exist" Oct 30 20:29:56.221367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.221335 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:29:56.270940 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:56.270915 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-w5-407632-kserve-b79fdcf4d-gv4sx"] Oct 30 20:29:57.091973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:57.091940 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:57.298020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:57.297989 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" path="/var/lib/kubelet/pods/6262a9d5-36ff-4999-b1d6-1191b8303485/volumes" Oct 30 20:29:59.677232 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:59.675230 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:59.883259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:29:59.883226 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:02.130656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:02.130626 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:06.979015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:06.978983 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:11.767073 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:11.767036 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.784177 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:12.784140 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:30:15.891676 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:15.891632 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:30:15.892121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:15.891912 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" podUID="a2547fe3-68f3-4141-88b5-091263f916db" containerName="maas-api" containerID="cri-o://c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c" gracePeriod=30 Oct 30 20:30:16.142454 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.142390 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:16.157321 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.157299 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:30:16.159852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.159824 4524 generic.go:356] "Generic (PLEG): container finished" podID="a2547fe3-68f3-4141-88b5-091263f916db" containerID="c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c" exitCode=0 Oct 30 20:30:16.159974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.159920 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" Oct 30 20:30:16.159974 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.159958 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" event={"ID":"a2547fe3-68f3-4141-88b5-091263f916db","Type":"ContainerDied","Data":"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c"} Oct 30 20:30:16.160102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.160004 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz" event={"ID":"a2547fe3-68f3-4141-88b5-091263f916db","Type":"ContainerDied","Data":"b842008004929debe220535f1fb412fa18784bb69062aeb6e15ae50964f45327"} Oct 30 20:30:16.160102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.160030 4524 scope.go:122] "RemoveContainer" containerID="c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c" Oct 30 20:30:16.170975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.170955 4524 scope.go:122] "RemoveContainer" containerID="c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c" Oct 30 20:30:16.171909 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:16.171884 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c\": container with ID starting with c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c not found: ID does not exist" containerID="c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c" Oct 30 20:30:16.172020 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.171921 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c"} err="failed to get container status \"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c\": rpc error: code = NotFound desc = could not find container \"c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c\": container with ID starting with c2ca29cbf806eedba7b0c85a04a4051e3d717cd06c8488fffde6ac48e7d3c53c not found: ID does not exist" Oct 30 20:30:16.241960 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.241917 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\") pod \"a2547fe3-68f3-4141-88b5-091263f916db\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " Oct 30 20:30:16.242120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.242016 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\") pod \"a2547fe3-68f3-4141-88b5-091263f916db\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " Oct 30 20:30:16.242120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.242040 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") pod \"a2547fe3-68f3-4141-88b5-091263f916db\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " Oct 30 20:30:16.242120 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.242105 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\" (UniqueName: \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\") pod \"a2547fe3-68f3-4141-88b5-091263f916db\" (UID: \"a2547fe3-68f3-4141-88b5-091263f916db\") " Oct 30 20:30:16.242444 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.242419 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca" pod "a2547fe3-68f3-4141-88b5-091263f916db" (UID: "a2547fe3-68f3-4141-88b5-091263f916db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:16.243971 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.243944 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs" pod "a2547fe3-68f3-4141-88b5-091263f916db" (UID: "a2547fe3-68f3-4141-88b5-091263f916db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:16.244135 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.244113 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls" pod "a2547fe3-68f3-4141-88b5-091263f916db" (UID: "a2547fe3-68f3-4141-88b5-091263f916db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:16.244183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.244116 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q" pod "a2547fe3-68f3-4141-88b5-091263f916db" (UID: "a2547fe3-68f3-4141-88b5-091263f916db"). InnerVolumeSpecName "kube-api-access-cnh8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:16.343255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.343214 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2547fe3-68f3-4141-88b5-091263f916db-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:16.343255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.343247 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:16.343255 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.343257 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnh8q\" (UniqueName: \"kubernetes.io/projected/a2547fe3-68f3-4141-88b5-091263f916db-kube-api-access-cnh8q\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:16.343555 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.343267 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2547fe3-68f3-4141-88b5-091263f916db-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:16.590856 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.590824 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:30:16.594523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:16.594487 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14c846-8769bd6c7-wllqz"] Oct 30 20:30:17.299034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:17.298992 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2547fe3-68f3-4141-88b5-091263f916db" path="/var/lib/kubelet/pods/a2547fe3-68f3-4141-88b5-091263f916db/volumes" Oct 30 20:30:17.782641 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:17.782606 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:18.164451 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:18.164414 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:18.507122 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:18.507012 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:18.727680 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:18.727640 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:18.969132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:18.969096 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:19.231866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:19.231793 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:19.829181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:19.829146 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:20.432287 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:20.432258 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:21.087348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:21.087313 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:22.467124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.467093 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 20:30:22.467535 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.467334 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" podUID="d842657a-9d13-4742-b136-adc407e85395" containerName="maas-api" containerID="cri-o://7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d" gracePeriod=30 Oct 30 20:30:22.623169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.623141 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:22.731724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.731658 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 20:30:22.817602 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.817569 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\") pod \"d842657a-9d13-4742-b136-adc407e85395\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " Oct 30 20:30:22.817784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.817616 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") pod \"d842657a-9d13-4742-b136-adc407e85395\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " Oct 30 20:30:22.817784 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.817651 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\" (UniqueName: \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\") pod \"d842657a-9d13-4742-b136-adc407e85395\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " Oct 30 20:30:22.817932 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.817787 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\") pod \"d842657a-9d13-4742-b136-adc407e85395\" (UID: \"d842657a-9d13-4742-b136-adc407e85395\") " Oct 30 20:30:22.818471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.818438 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca" pod "d842657a-9d13-4742-b136-adc407e85395" (UID: "d842657a-9d13-4742-b136-adc407e85395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:22.819981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.819951 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls" pod "d842657a-9d13-4742-b136-adc407e85395" (UID: "d842657a-9d13-4742-b136-adc407e85395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:22.820098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.820047 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw" pod "d842657a-9d13-4742-b136-adc407e85395" (UID: "d842657a-9d13-4742-b136-adc407e85395"). InnerVolumeSpecName "kube-api-access-qs2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:22.820098 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.820092 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs" pod "d842657a-9d13-4742-b136-adc407e85395" (UID: "d842657a-9d13-4742-b136-adc407e85395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:22.918650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.918614 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d842657a-9d13-4742-b136-adc407e85395-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:22.918650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.918645 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:22.918650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.918654 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d842657a-9d13-4742-b136-adc407e85395-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:22.918908 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:22.918663 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs2tw\" (UniqueName: \"kubernetes.io/projected/d842657a-9d13-4742-b136-adc407e85395-kube-api-access-qs2tw\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.194426 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.194391 4524 generic.go:356] "Generic (PLEG): container finished" podID="d842657a-9d13-4742-b136-adc407e85395" containerID="7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d" exitCode=0 Oct 30 20:30:23.194571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.194443 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" event={"ID":"d842657a-9d13-4742-b136-adc407e85395","Type":"ContainerDied","Data":"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d"} Oct 30 20:30:23.194571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.194450 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" Oct 30 20:30:23.194571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.194513 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx" event={"ID":"d842657a-9d13-4742-b136-adc407e85395","Type":"ContainerDied","Data":"2f710a0913670dd52b392ec3c18221e618b6007b10abbbb7babfe9670e5b9d83"} Oct 30 20:30:23.194571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.194537 4524 scope.go:122] "RemoveContainer" containerID="7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d" Oct 30 20:30:23.203509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.203488 4524 scope.go:122] "RemoveContainer" containerID="7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d" Oct 30 20:30:23.203877 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:23.203853 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d\": container with ID starting with 7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d not found: ID does not exist" containerID="7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d" Oct 30 20:30:23.203981 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.203883 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d"} err="failed to get container status \"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d\": rpc error: code = NotFound desc = could not find container \"7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d\": container with ID starting with 7c5a64c1c6f21119f040451e45aa00e541d2da513477a130081fa1700a09938d not found: ID does not exist" Oct 30 20:30:23.360046 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.360017 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 20:30:23.412022 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:23.411988 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-407632-7b7b5c9db5-54sjx"] Oct 30 20:30:24.152778 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:24.152740 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:24.402724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:24.399415 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:24.614288 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:24.614198 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:24.836571 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:24.836287 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:25.085654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.085619 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:25.298546 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.298506 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d842657a-9d13-4742-b136-adc407e85395" path="/var/lib/kubelet/pods/d842657a-9d13-4742-b136-adc407e85395/volumes" Oct 30 20:30:25.522918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.522882 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:25.919876 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.919837 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:25.956632 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.956600 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:30:25.956863 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:25.956839 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" containerID="cri-o://f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12" gracePeriod=30 Oct 30 20:30:26.170099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:26.170002 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:28.091438 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:28.091404 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:28.343629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:28.343552 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:29.826550 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:29.826508 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:31.212799 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.212778 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:30:31.235067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.235012 4524 generic.go:356] "Generic (PLEG): container finished" podID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerID="f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12" exitCode=0 Oct 30 20:30:31.235067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.235048 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerDied","Data":"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12"} Oct 30 20:30:31.235067 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.235056 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" Oct 30 20:30:31.235250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.235094 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf" event={"ID":"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88","Type":"ContainerDied","Data":"353e8a9ff87e3619fb1bb976eec52d53dcf4a23efe27f5faa80713aad3eab490"} Oct 30 20:30:31.235250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.235118 4524 scope.go:122] "RemoveContainer" containerID="f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12" Oct 30 20:30:31.243144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.243127 4524 scope.go:122] "RemoveContainer" containerID="8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004" Oct 30 20:30:31.250720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.250683 4524 scope.go:122] "RemoveContainer" containerID="f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12" Oct 30 20:30:31.250970 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:31.250950 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12\": container with ID starting with f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12 not found: ID does not exist" containerID="f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12" Oct 30 20:30:31.251037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.250982 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12"} err="failed to get container status \"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12\": rpc error: code = NotFound desc = could not find container \"f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12\": container with ID starting with f3bc5ab0fd0d21ec633a716ce743428155c3316681a9c362ce7c0db768b0fd12 not found: ID does not exist" Oct 30 20:30:31.251037 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.251001 4524 scope.go:122] "RemoveContainer" containerID="8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004" Oct 30 20:30:31.251271 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:31.251251 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004\": container with ID starting with 8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004 not found: ID does not exist" containerID="8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004" Oct 30 20:30:31.251320 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.251277 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004"} err="failed to get container status \"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004\": rpc error: code = NotFound desc = could not find container \"8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004\": container with ID starting with 8920a9b25fc72d78d05a19726b580f64c7df3bb8bf7ea8c5eb98889a3e1cf004 not found: ID does not exist" Oct 30 20:30:31.402586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402552 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402586 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402629 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402656 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402687 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402746 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.402848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402837 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.403140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402879 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.403140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402925 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\") pod \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\" (UID: \"2d8bf821-f4ce-453c-934e-4d9a6b9fbe88\") " Oct 30 20:30:31.403140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402973 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:31.403140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.402989 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:31.403356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.403228 4524 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-credential-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.403356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.403235 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:31.403356 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.403246 4524 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-data\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.403522 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.403488 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:31.403589 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.403558 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:31.405201 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.405172 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:31.405345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.405247 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:31.405513 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.405497 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "kube-api-access-h5rf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:31.405580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.405561 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo" pod "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" (UID: "2d8bf821-f4ce-453c-934e-4d9a6b9fbe88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:30:31.504031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.503974 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5rf9\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-kube-api-access-h5rf9\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.503998 4524 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istiod-ca-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.504007 4524 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-token\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.504018 4524 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-socket\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504031 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.504028 4524 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-workload-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.504037 4524 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-podinfo\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.504264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.504046 4524 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88-istio-envoy\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:31.655269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.655241 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:30:31.662004 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:31.661982 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-14c846-openshift-default-556f8d5875-lssrf"] Oct 30 20:30:32.767897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:32.767860 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.033923 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:33.033849 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.275483 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:33.275449 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.298050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:33.297977 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" path="/var/lib/kubelet/pods/2d8bf821-f4ce-453c-934e-4d9a6b9fbe88/volumes" Oct 30 20:30:34.723765 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:34.723723 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:38.339333 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:38.339300 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:30:38.666048 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:38.666007 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:38.884267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:38.884239 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.611868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:39.611832 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:42.784845 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:42.784807 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:30:43.340203 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:43.340167 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" containerID="cri-o://14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4" gracePeriod=25 Oct 30 20:30:44.045310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:44.045273 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:44.458805 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:44.458769 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.529557 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.529521 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:30:48.530080 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.529773 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" podUID="3a33b4c9-1302-47dc-972a-211dd750410b" containerName="maas-api" containerID="cri-o://2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6" gracePeriod=30 Oct 30 20:30:48.725759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.725680 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.784496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.784439 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:30:48.857345 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.857314 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\") pod \"3a33b4c9-1302-47dc-972a-211dd750410b\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " Oct 30 20:30:48.857521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.857400 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\") pod \"3a33b4c9-1302-47dc-972a-211dd750410b\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " Oct 30 20:30:48.857521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.857461 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") pod \"3a33b4c9-1302-47dc-972a-211dd750410b\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " Oct 30 20:30:48.857521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.857490 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\" (UniqueName: \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\") pod \"3a33b4c9-1302-47dc-972a-211dd750410b\" (UID: \"3a33b4c9-1302-47dc-972a-211dd750410b\") " Oct 30 20:30:48.857767 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.857737 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca" pod "3a33b4c9-1302-47dc-972a-211dd750410b" (UID: "3a33b4c9-1302-47dc-972a-211dd750410b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:48.859849 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.859823 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw" pod "3a33b4c9-1302-47dc-972a-211dd750410b" (UID: "3a33b4c9-1302-47dc-972a-211dd750410b"). InnerVolumeSpecName "kube-api-access-dm4qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:48.860063 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.860035 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs" pod "3a33b4c9-1302-47dc-972a-211dd750410b" (UID: "3a33b4c9-1302-47dc-972a-211dd750410b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:48.860256 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.860232 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls" pod "3a33b4c9-1302-47dc-972a-211dd750410b" (UID: "3a33b4c9-1302-47dc-972a-211dd750410b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:48.958932 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.958896 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:48.958932 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.958926 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a33b4c9-1302-47dc-972a-211dd750410b-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:48.959136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.958940 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4qw\" (UniqueName: \"kubernetes.io/projected/3a33b4c9-1302-47dc-972a-211dd750410b-kube-api-access-dm4qw\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:48.959136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:48.958955 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a33b4c9-1302-47dc-972a-211dd750410b-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:49.321245 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.321215 4524 generic.go:356] "Generic (PLEG): container finished" podID="3a33b4c9-1302-47dc-972a-211dd750410b" containerID="2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6" exitCode=0 Oct 30 20:30:49.321409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.321261 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" Oct 30 20:30:49.321409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.321266 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" event={"ID":"3a33b4c9-1302-47dc-972a-211dd750410b","Type":"ContainerDied","Data":"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6"} Oct 30 20:30:49.321409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.321384 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58" event={"ID":"3a33b4c9-1302-47dc-972a-211dd750410b","Type":"ContainerDied","Data":"8090a0e7011d36975f45f2a62903c11c3d62a9627d8f095eec9f9a9ffd7a7a5d"} Oct 30 20:30:49.321409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.321412 4524 scope.go:122] "RemoveContainer" containerID="2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6" Oct 30 20:30:49.330470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.330451 4524 scope.go:122] "RemoveContainer" containerID="2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6" Oct 30 20:30:49.330778 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:30:49.330760 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6\": container with ID starting with 2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6 not found: ID does not exist" containerID="2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6" Oct 30 20:30:49.330871 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.330785 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6"} err="failed to get container status \"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6\": rpc error: code = NotFound desc = could not find container \"2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6\": container with ID starting with 2a0502a58d4ee5455ff4883b24063550f4f90a5cadd8dcc9b2083ae6e7a6fbe6 not found: ID does not exist" Oct 30 20:30:49.487375 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.487342 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:30:49.492064 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:49.492038 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72db69-c55fdc488-pzz58"] Oct 30 20:30:50.230852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:50.230809 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.444040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:50.444009 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.656482 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:50.656447 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:51.200547 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:51.200511 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:51.299181 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:51.299147 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a33b4c9-1302-47dc-972a-211dd750410b" path="/var/lib/kubelet/pods/3a33b4c9-1302-47dc-972a-211dd750410b/volumes" Oct 30 20:30:51.434102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:51.434065 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:54.044803 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:54.044764 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:54.751146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:54.751109 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:54.969881 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:54.969851 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:55.464975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:55.464908 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.932569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:57.932533 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.161013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:58.160980 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.417628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:58.417598 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.833477 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:30:58.833378 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.524305 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:00.524278 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:03.594656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.594632 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:31:03.695021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.694994 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\") pod \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " Oct 30 20:31:03.695179 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.695040 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\") pod \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\" (UID: \"4fbe29f2-115b-4722-a144-fa1c981cd0a4\") " Oct 30 20:31:03.695431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.695409 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume" pod "4fbe29f2-115b-4722-a144-fa1c981cd0a4" (UID: "4fbe29f2-115b-4722-a144-fa1c981cd0a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:03.697099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.697074 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4" pod "4fbe29f2-115b-4722-a144-fa1c981cd0a4" (UID: "4fbe29f2-115b-4722-a144-fa1c981cd0a4"). InnerVolumeSpecName "kube-api-access-xqhp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:03.796143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.796113 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbe29f2-115b-4722-a144-fa1c981cd0a4-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:03.796143 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.796141 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/4fbe29f2-115b-4722-a144-fa1c981cd0a4-kube-api-access-xqhp4\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:03.906450 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:03.906418 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:04.391953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.391913 4524 generic.go:356] "Generic (PLEG): container finished" podID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerID="14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4" exitCode=0 Oct 30 20:31:04.392133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.391962 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" Oct 30 20:31:04.392133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.392000 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" event={"ID":"4fbe29f2-115b-4722-a144-fa1c981cd0a4","Type":"ContainerDied","Data":"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4"} Oct 30 20:31:04.392133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.392059 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g" event={"ID":"4fbe29f2-115b-4722-a144-fa1c981cd0a4","Type":"ContainerDied","Data":"3df2917f2144b9a593c186d100a6665f80306b15e68f221206ffcc72c6c8c75f"} Oct 30 20:31:04.392133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.392084 4524 scope.go:122] "RemoveContainer" containerID="14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4" Oct 30 20:31:04.400730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.400710 4524 scope.go:122] "RemoveContainer" containerID="14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4" Oct 30 20:31:04.401031 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:31:04.401008 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4\": container with ID starting with 14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4 not found: ID does not exist" containerID="14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4" Oct 30 20:31:04.401088 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.401042 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4"} err="failed to get container status \"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4\": rpc error: code = NotFound desc = could not find container \"14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4\": container with ID starting with 14e273840aaa25d696aad6b571574d855c8bb8b0a92bc09425ee260f4cd873b4 not found: ID does not exist" Oct 30 20:31:04.557363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.557333 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:31:04.606811 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:04.606785 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72db69-69b77848c5-8lg4g"] Oct 30 20:31:05.298265 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:05.298227 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" path="/var/lib/kubelet/pods/4fbe29f2-115b-4722-a144-fa1c981cd0a4/volumes" Oct 30 20:31:05.474024 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:05.473994 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:11.208367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:11.208327 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:11.427043 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:11.427006 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.783561 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:31:12.783529 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:31:24.392290 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.392251 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 20:31:24.392797 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.392478 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" podUID="a0e02b61-5689-482a-aeb4-544bd5e88d50" containerName="maas-api" containerID="cri-o://b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc" gracePeriod=30 Oct 30 20:31:24.643060 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.643006 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 20:31:24.666269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.666241 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:24.671259 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.671229 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\") pod \"a0e02b61-5689-482a-aeb4-544bd5e88d50\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " Oct 30 20:31:24.671389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.671317 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\" (UniqueName: \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\") pod \"a0e02b61-5689-482a-aeb4-544bd5e88d50\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " Oct 30 20:31:24.671389 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.671362 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\") pod \"a0e02b61-5689-482a-aeb4-544bd5e88d50\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " Oct 30 20:31:24.671518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.671403 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\") pod \"a0e02b61-5689-482a-aeb4-544bd5e88d50\" (UID: \"a0e02b61-5689-482a-aeb4-544bd5e88d50\") " Oct 30 20:31:24.671938 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.671898 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca" pod "a0e02b61-5689-482a-aeb4-544bd5e88d50" (UID: "a0e02b61-5689-482a-aeb4-544bd5e88d50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:24.673901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.673856 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls" pod "a0e02b61-5689-482a-aeb4-544bd5e88d50" (UID: "a0e02b61-5689-482a-aeb4-544bd5e88d50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:24.674034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.673947 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z" pod "a0e02b61-5689-482a-aeb4-544bd5e88d50" (UID: "a0e02b61-5689-482a-aeb4-544bd5e88d50"). InnerVolumeSpecName "kube-api-access-6922z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:24.674217 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.674183 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs" pod "a0e02b61-5689-482a-aeb4-544bd5e88d50" (UID: "a0e02b61-5689-482a-aeb4-544bd5e88d50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:24.772272 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.772236 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:24.772272 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.772267 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e02b61-5689-482a-aeb4-544bd5e88d50-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:24.772488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.772284 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6922z\" (UniqueName: \"kubernetes.io/projected/a0e02b61-5689-482a-aeb4-544bd5e88d50-kube-api-access-6922z\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:24.772488 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:24.772297 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e02b61-5689-482a-aeb4-544bd5e88d50-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:25.493751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.493716 4524 generic.go:356] "Generic (PLEG): container finished" podID="a0e02b61-5689-482a-aeb4-544bd5e88d50" containerID="b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc" exitCode=0 Oct 30 20:31:25.494243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.493764 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" Oct 30 20:31:25.494243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.493787 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" event={"ID":"a0e02b61-5689-482a-aeb4-544bd5e88d50","Type":"ContainerDied","Data":"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc"} Oct 30 20:31:25.494243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.493829 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb" event={"ID":"a0e02b61-5689-482a-aeb4-544bd5e88d50","Type":"ContainerDied","Data":"0c33e7d01fe51835eced3958d74b78451904f455b47d4b9f7bcf8fc3118f2599"} Oct 30 20:31:25.494243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.493852 4524 scope.go:122] "RemoveContainer" containerID="b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc" Oct 30 20:31:25.509144 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.509120 4524 scope.go:122] "RemoveContainer" containerID="b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc" Oct 30 20:31:25.509536 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:31:25.509509 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc\": container with ID starting with b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc not found: ID does not exist" containerID="b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc" Oct 30 20:31:25.509657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.509548 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc"} err="failed to get container status \"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc\": rpc error: code = NotFound desc = could not find container \"b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc\": container with ID starting with b21a94ca05c81f7d3e0c7411236f76628b87e65cf14b03671c4cc646a2519bfc not found: ID does not exist" Oct 30 20:31:25.660891 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.660866 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 20:31:25.710825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:25.710799 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3ccc9f-5546d4bf6b-5lvdb"] Oct 30 20:31:26.229396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:26.229347 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:26.493655 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:26.493579 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:27.192980 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:27.192946 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:27.297543 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:27.297513 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e02b61-5689-482a-aeb4-544bd5e88d50" path="/var/lib/kubelet/pods/a0e02b61-5689-482a-aeb4-544bd5e88d50/volumes" Oct 30 20:31:34.234447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:34.234412 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:34.503188 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:34.503078 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:34.745093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:34.745057 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:40.025893 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.025856 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 20:31:40.026348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.026057 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" podUID="57065cb7-fc39-4f4b-8799-43c432f4dc83" containerName="manager" containerID="cri-o://ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee" gracePeriod=30 Oct 30 20:31:40.281370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.281313 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 20:31:40.400402 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.400370 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\" (UniqueName: \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\") pod \"57065cb7-fc39-4f4b-8799-43c432f4dc83\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " Oct 30 20:31:40.400537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.400412 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\") pod \"57065cb7-fc39-4f4b-8799-43c432f4dc83\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " Oct 30 20:31:40.400537 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.400476 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\") pod \"57065cb7-fc39-4f4b-8799-43c432f4dc83\" (UID: \"57065cb7-fc39-4f4b-8799-43c432f4dc83\") " Oct 30 20:31:40.403376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.402967 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn" pod "57065cb7-fc39-4f4b-8799-43c432f4dc83" (UID: "57065cb7-fc39-4f4b-8799-43c432f4dc83"). InnerVolumeSpecName "kube-api-access-6rxgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:40.403376 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.403244 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert" pod "57065cb7-fc39-4f4b-8799-43c432f4dc83" (UID: "57065cb7-fc39-4f4b-8799-43c432f4dc83"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:40.403748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.403721 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs" pod "57065cb7-fc39-4f4b-8799-43c432f4dc83" (UID: "57065cb7-fc39-4f4b-8799-43c432f4dc83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:40.501445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.501412 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rxgn\" (UniqueName: \"kubernetes.io/projected/57065cb7-fc39-4f4b-8799-43c432f4dc83-kube-api-access-6rxgn\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:40.501445 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.501444 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:40.501647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.501459 4524 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57065cb7-fc39-4f4b-8799-43c432f4dc83-webhook-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:40.563576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.563490 4524 generic.go:356] "Generic (PLEG): container finished" podID="57065cb7-fc39-4f4b-8799-43c432f4dc83" containerID="ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee" exitCode=0 Oct 30 20:31:40.563576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.563546 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" Oct 30 20:31:40.563835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.563577 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" event={"ID":"57065cb7-fc39-4f4b-8799-43c432f4dc83","Type":"ContainerDied","Data":"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee"} Oct 30 20:31:40.563835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.563646 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-qz5rt" event={"ID":"57065cb7-fc39-4f4b-8799-43c432f4dc83","Type":"ContainerDied","Data":"f6a9e8f70407d62ba0c3ab95135347afbd7eed4e3af3af50d08c06b120dcba49"} Oct 30 20:31:40.563835 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.563676 4524 scope.go:122] "RemoveContainer" containerID="ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee" Oct 30 20:31:40.573749 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.573730 4524 scope.go:122] "RemoveContainer" containerID="ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee" Oct 30 20:31:40.574001 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:31:40.573982 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee\": container with ID starting with ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee not found: ID does not exist" containerID="ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee" Oct 30 20:31:40.574055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.574011 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee"} err="failed to get container status \"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee\": rpc error: code = NotFound desc = could not find container \"ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee\": container with ID starting with ea703ebb3d6ad6aa9601f840cd265c3d2b052d62e38ce1b106c11dbbc58eadee not found: ID does not exist" Oct 30 20:31:40.729855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.729820 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 20:31:40.733678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:40.733650 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-qz5rt"] Oct 30 20:31:41.298075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.298045 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57065cb7-fc39-4f4b-8799-43c432f4dc83" path="/var/lib/kubelet/pods/57065cb7-fc39-4f4b-8799-43c432f4dc83/volumes" Oct 30 20:31:41.549648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.549567 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:41.644380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644342 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:31:41.644897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644869 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.644897 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644892 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644905 4524 cpu_manager.go:414] "containerMap: removing container" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644914 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644931 4524 cpu_manager.go:414] "containerMap: removing container" podUID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644938 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644951 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644959 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644970 4524 cpu_manager.go:414] "containerMap: removing container" podUID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644979 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644991 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.644998 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645010 4524 cpu_manager.go:414] "containerMap: removing container" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645017 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645033 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d842657a-9d13-4742-b136-adc407e85395" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645040 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d842657a-9d13-4742-b136-adc407e85395" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645053 4524 cpu_manager.go:414] "containerMap: removing container" podUID="57065cb7-fc39-4f4b-8799-43c432f4dc83" containerName="manager" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645061 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57065cb7-fc39-4f4b-8799-43c432f4dc83" containerName="manager" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645070 4524 cpu_manager.go:414] "containerMap: removing container" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645077 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645086 4524 cpu_manager.go:414] "containerMap: removing container" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645093 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645104 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3a33b4c9-1302-47dc-972a-211dd750410b" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645111 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a33b4c9-1302-47dc-972a-211dd750410b" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645124 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645131 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645142 4524 cpu_manager.go:414] "containerMap: removing container" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645149 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645164 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645171 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645186 4524 cpu_manager.go:414] "containerMap: removing container" podUID="bce877b5-0218-4c55-8d5d-25083e1894ea" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645192 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce877b5-0218-4c55-8d5d-25083e1894ea" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645203 4524 cpu_manager.go:414] "containerMap: removing container" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645210 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645223 4524 cpu_manager.go:414] "containerMap: removing container" podUID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645229 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" containerName="maas-api" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645241 4524 cpu_manager.go:414] "containerMap: removing container" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" Oct 30 20:31:41.645220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645248 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645258 4524 cpu_manager.go:414] "containerMap: removing container" podUID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645265 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645278 4524 cpu_manager.go:414] "containerMap: removing container" podUID="a0e02b61-5689-482a-aeb4-544bd5e88d50" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645284 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e02b61-5689-482a-aeb4-544bd5e88d50" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645296 4524 cpu_manager.go:414] "containerMap: removing container" podUID="a2547fe3-68f3-4141-88b5-091263f916db" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645301 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2547fe3-68f3-4141-88b5-091263f916db" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645311 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645318 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645331 4524 cpu_manager.go:414] "containerMap: removing container" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645338 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645347 4524 cpu_manager.go:414] "containerMap: removing container" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerName="payload-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645354 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerName="payload-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645434 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645446 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645455 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce877b5-0218-4c55-8d5d-25083e1894ea" containerName="main" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645464 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3896eeb3-6acb-424e-954b-9d5e35634d7b" containerName="main" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645475 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5f5ae7-0edb-40e5-a1aa-8190b3c8a212" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645485 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="871b5b61-8f4a-4e16-bff7-f18801b67b21" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645493 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c8ea09-5ca6-412f-bda8-63a5e68a1a25" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645501 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645511 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e02b61-5689-482a-aeb4-544bd5e88d50" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645520 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2547fe3-68f3-4141-88b5-091263f916db" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645528 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbe29f2-115b-4722-a144-fa1c981cd0a4" containerName="payload-pre-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645538 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645545 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c1c650-6d4c-45cb-b5f2-56e523b54365" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645553 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd70019-8732-45c7-a432-9d78df73b7ba" containerName="payload-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645563 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645571 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645579 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a9f0ac-f3aa-417a-9545-0ed992a1bfa8" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645588 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8bf821-f4ce-453c-934e-4d9a6b9fbe88" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645596 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d788ee1-cd0c-486c-9ee4-3bd725f1433b" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645605 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b6ca18-647d-4394-a749-10fba8eb4be3" containerName="payload-pre-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645612 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3870242-9d90-4662-8a01-d06fe5defbc1" containerName="payload-pre-processing" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645622 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d842657a-9d13-4742-b136-adc407e85395" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645630 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="57065cb7-fc39-4f4b-8799-43c432f4dc83" containerName="manager" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645638 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="6262a9d5-36ff-4999-b1d6-1191b8303485" containerName="main" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645645 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="321a585b-81da-4d31-af5b-4de25af3e1fb" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645653 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a33b4c9-1302-47dc-972a-211dd750410b" containerName="maas-api" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645661 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:31:41.647175 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.645669 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e262ae1-982f-4de1-9f21-422fa840d6bd" containerName="istio-proxy" Oct 30 20:31:41.650202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.650182 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.712183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.712149 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.712338 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.712276 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgjd\" (UniqueName: \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.712338 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.712311 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.753408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.753388 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:31:41.753578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.753391 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wdmrl\"" Oct 30 20:31:41.753578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.753430 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:31:41.753732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.753546 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:31:41.755161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.755138 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:31:41.759033 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.759005 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:41.813601 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.813542 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.813720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.813601 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.813720 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.813661 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgjd\" (UniqueName: \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.816163 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.816137 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.816270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.816136 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.875077 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.875047 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgjd\" (UniqueName: \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\") pod \"maas-controller-84c66fdd7b-p8687\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:41.962661 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:41.962631 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:42.311506 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:42.311472 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:42.336496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:42.336354 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:31:42.339232 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:31:42.339192 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d78496d_ec7e_4ad8_b17a_06c82a286c7a.slice/crio-c3bb74c38ff752b36c9d895216d705a6267f0d5bc396e24eda29f7d504ebb928 WatchSource:0}: Error finding container c3bb74c38ff752b36c9d895216d705a6267f0d5bc396e24eda29f7d504ebb928: Status 404 returned error can't find the container with id c3bb74c38ff752b36c9d895216d705a6267f0d5bc396e24eda29f7d504ebb928 Oct 30 20:31:42.340536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:42.340517 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:31:42.575405 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:42.575321 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-p8687" event={"ID":"2d78496d-ec7e-4ad8-b17a-06c82a286c7a","Type":"ContainerStarted","Data":"c3bb74c38ff752b36c9d895216d705a6267f0d5bc396e24eda29f7d504ebb928"} Oct 30 20:31:42.784135 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:31:42.784087 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:31:43.581831 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:43.581795 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-p8687" event={"ID":"2d78496d-ec7e-4ad8-b17a-06c82a286c7a","Type":"ContainerStarted","Data":"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476"} Oct 30 20:31:43.582253 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:43.581958 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:31:43.700901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:43.700854 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-84c66fdd7b-p8687" podStartSLOduration=1.6531361439999999 podStartE2EDuration="2.70083848s" totalImagesPullingTime="1.047702336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:43.699952537 +0000 UTC m=+2998.543624168" watchObservedRunningTime="2026-10-30 20:31:43.70083848 +0000 UTC m=+2998.544510108" Oct 30 20:31:48.724036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:48.724001 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.585735 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:51.585677 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:54.589140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:31:54.589105 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:32:00.147678 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:00.147596 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:00.376654 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:00.376625 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:01.867136 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867072 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz"] Oct 30 20:32:01.867670 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867646 4524 cpu_manager.go:414] "containerMap: removing container" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:32:01.867732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867675 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8286b115-c193-457f-8882-bf43b1fe4b3c" containerName="istio-proxy" Oct 30 20:32:01.867732 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867727 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:32:01.867807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867736 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5238180-2ad7-4b20-bec7-5c244c68c2d0" containerName="istio-proxy" Oct 30 20:32:01.867807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867775 4524 cpu_manager.go:414] "containerMap: removing container" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:32:01.867807 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.867784 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463488c8-7f01-4773-a3c8-72dc66418507" containerName="istio-proxy" Oct 30 20:32:01.872115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.872094 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.924996 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.924968 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\"" Oct 30 20:32:01.977130 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.977101 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz"] Oct 30 20:32:01.985202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985181 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985214 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985235 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985370 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985295 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985400 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05436fec-ac75-45e2-83aa-90eda66455dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985443 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985429 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05436fec-ac75-45e2-83aa-90eda66455dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985501 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnkh\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-kube-api-access-ttnkh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985607 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985590 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:01.985745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:01.985728 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086774 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086732 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086814 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086834 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086856 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086880 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086910 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.086968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086938 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05436fec-ac75-45e2-83aa-90eda66455dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.086981 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05436fec-ac75-45e2-83aa-90eda66455dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087250 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087008 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnkh\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-kube-api-access-ttnkh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087254 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087362 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087301 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087356 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087526 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087498 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.087901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.087872 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05436fec-ac75-45e2-83aa-90eda66455dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.089526 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.089501 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05436fec-ac75-45e2-83aa-90eda66455dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.089841 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.089819 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05436fec-ac75-45e2-83aa-90eda66455dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.158935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.158902 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnkh\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-kube-api-access-ttnkh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.161060 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.161035 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05436fec-ac75-45e2-83aa-90eda66455dc-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-crkfz\" (UID: \"05436fec-ac75-45e2-83aa-90eda66455dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.184927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.184902 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:02.615192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.615112 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz"] Oct 30 20:32:02.617481 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:32:02.617448 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05436fec_ac75_45e2_83aa_90eda66455dc.slice/crio-4f04efe04820dfc21c9c3008f6b770822faa8ee0c51eb7b551715b49c537f287 WatchSource:0}: Error finding container 4f04efe04820dfc21c9c3008f6b770822faa8ee0c51eb7b551715b49c537f287: Status 404 returned error can't find the container with id 4f04efe04820dfc21c9c3008f6b770822faa8ee0c51eb7b551715b49c537f287 Oct 30 20:32:02.619634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.619605 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:02.619693 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.619669 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:02.619759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.619745 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:02.662408 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:02.662378 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" event={"ID":"05436fec-ac75-45e2-83aa-90eda66455dc","Type":"ContainerStarted","Data":"4f04efe04820dfc21c9c3008f6b770822faa8ee0c51eb7b551715b49c537f287"} Oct 30 20:32:03.678578 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:03.678541 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" event={"ID":"05436fec-ac75-45e2-83aa-90eda66455dc","Type":"ContainerStarted","Data":"c653482e59c71c6163dfe3b5c966642bd340ccbc060b82fde06c35922c3f0a75"} Oct 30 20:32:03.802309 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:03.802246 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podStartSLOduration=2.802230492 podStartE2EDuration="2.802230492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:03.80024548 +0000 UTC m=+3018.643917110" watchObservedRunningTime="2026-10-30 20:32:03.802230492 +0000 UTC m=+3018.645902121" Oct 30 20:32:04.185276 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:04.185232 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:04.186922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:04.186898 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:04.186984 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:04.186951 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:05.185988 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:05.185950 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:05.186401 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:05.186003 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:06.185922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:06.185889 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:06.186148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:06.185946 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:07.185785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:07.185753 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:07.185965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:07.185805 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:08.185941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:08.185905 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:08.186434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:08.185971 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:09.185913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:09.185874 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:32:09.186128 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:09.185932 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:32:11.186758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:11.186719 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:32:11.187299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:11.186776 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:32:12.228845 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:12.228815 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:32:12.229244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:12.228919 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:32:12.788520 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:32:12.788477 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:32:13.013647 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:13.013611 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:13.014015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:13.013979 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:13.014802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:13.014787 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" Oct 30 20:32:16.172776 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.172744 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:16.250481 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.250451 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:32:16.250655 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.250635 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-84c66fdd7b-p8687" podUID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" containerName="manager" containerID="cri-o://2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476" gracePeriod=30 Oct 30 20:32:16.491415 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.491390 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:32:16.632905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.631470 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\") pod \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " Oct 30 20:32:16.632905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.631664 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\" (UniqueName: \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\") pod \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " Oct 30 20:32:16.632905 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.631751 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\") pod \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\" (UID: \"2d78496d-ec7e-4ad8-b17a-06c82a286c7a\") " Oct 30 20:32:16.641821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.641777 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert" pod "2d78496d-ec7e-4ad8-b17a-06c82a286c7a" (UID: "2d78496d-ec7e-4ad8-b17a-06c82a286c7a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:16.641969 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.641837 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs" pod "2d78496d-ec7e-4ad8-b17a-06c82a286c7a" (UID: "2d78496d-ec7e-4ad8-b17a-06c82a286c7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:16.642115 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.642094 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd" pod "2d78496d-ec7e-4ad8-b17a-06c82a286c7a" (UID: "2d78496d-ec7e-4ad8-b17a-06c82a286c7a"). InnerVolumeSpecName "kube-api-access-9vgjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:16.733715 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.733623 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgjd\" (UniqueName: \"kubernetes.io/projected/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-kube-api-access-9vgjd\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.733715 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.733655 4524 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-webhook-cert\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.733715 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.733665 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d78496d-ec7e-4ad8-b17a-06c82a286c7a-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.740363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.740334 4524 generic.go:356] "Generic (PLEG): container finished" podID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" containerID="2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476" exitCode=0 Oct 30 20:32:16.740517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.740374 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-p8687" event={"ID":"2d78496d-ec7e-4ad8-b17a-06c82a286c7a","Type":"ContainerDied","Data":"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476"} Oct 30 20:32:16.740517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.740394 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-p8687" Oct 30 20:32:16.740517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.740457 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-p8687" event={"ID":"2d78496d-ec7e-4ad8-b17a-06c82a286c7a","Type":"ContainerDied","Data":"c3bb74c38ff752b36c9d895216d705a6267f0d5bc396e24eda29f7d504ebb928"} Oct 30 20:32:16.740517 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.740488 4524 scope.go:122] "RemoveContainer" containerID="2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476" Oct 30 20:32:16.749055 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.749038 4524 scope.go:122] "RemoveContainer" containerID="2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476" Oct 30 20:32:16.749307 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:32:16.749290 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476\": container with ID starting with 2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476 not found: ID does not exist" containerID="2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476" Oct 30 20:32:16.749372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.749312 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476"} err="failed to get container status \"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476\": rpc error: code = NotFound desc = could not find container \"2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476\": container with ID starting with 2bae08c9d606116f129d36a70124582acff761d1677841470b6aee830167c476 not found: ID does not exist" Oct 30 20:32:16.904823 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.904800 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:32:16.908792 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:16.908763 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-p8687"] Oct 30 20:32:17.003951 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.003890 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl"] Oct 30 20:32:17.004262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.004248 4524 cpu_manager.go:414] "containerMap: removing container" podUID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" containerName="manager" Oct 30 20:32:17.004303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.004264 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" containerName="manager" Oct 30 20:32:17.004339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.004322 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" containerName="manager" Oct 30 20:32:17.008543 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.008526 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.097053 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.097023 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl"] Oct 30 20:32:17.136719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136660 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b4a73f-b578-44ea-b4f5-a55650ea6011-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.136719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136713 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzrn\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-kube-api-access-6rzrn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.136937 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136805 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.136937 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136855 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.136937 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136909 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.137112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.136978 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.137112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.137012 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.137112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.137101 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.137246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.137145 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238301 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238276 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238314 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b4a73f-b578-44ea-b4f5-a55650ea6011-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238339 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzrn\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-kube-api-access-6rzrn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238362 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238384 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238506 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238581 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238612 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238665 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.238772 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238758 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.239289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.238915 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.239289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.239028 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.239289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.239085 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.239289 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.239211 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b4a73f-b578-44ea-b4f5-a55650ea6011-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.240766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.240743 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.241015 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.240996 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.299262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.299190 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d78496d-ec7e-4ad8-b17a-06c82a286c7a" path="/var/lib/kubelet/pods/2d78496d-ec7e-4ad8-b17a-06c82a286c7a/volumes" Oct 30 20:32:17.299954 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.299930 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.300169 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.300147 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzrn\" (UniqueName: \"kubernetes.io/projected/06b4a73f-b578-44ea-b4f5-a55650ea6011-kube-api-access-6rzrn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-jz4wl\" (UID: \"06b4a73f-b578-44ea-b4f5-a55650ea6011\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.321065 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.321046 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:17.744332 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.744304 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl"] Oct 30 20:32:17.747082 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:32:17.747054 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b4a73f_b578_44ea_b4f5_a55650ea6011.slice/crio-e95361b5f112de7a99a4de79874c48e73be75c14ebf394de10c264658ae4ecf8 WatchSource:0}: Error finding container e95361b5f112de7a99a4de79874c48e73be75c14ebf394de10c264658ae4ecf8: Status 404 returned error can't find the container with id e95361b5f112de7a99a4de79874c48e73be75c14ebf394de10c264658ae4ecf8 Oct 30 20:32:17.749150 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.749118 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:17.749270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.749193 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:17.749270 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:17.749240 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:18.268719 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.266679 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:18.409405 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.409367 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-kphzc"] Oct 30 20:32:18.413303 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.413274 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.472898 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.472859 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:32:18.473108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.472921 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wdmrl\"" Oct 30 20:32:18.473108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.473022 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:32:18.473108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.473056 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:32:18.474638 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.474615 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-kphzc"] Oct 30 20:32:18.574079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.573977 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-metrics-certs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.574249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.574152 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-webhook-cert\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.574307 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.574265 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftscs\" (UniqueName: \"kubernetes.io/projected/fce7e914-5b00-4b9a-905f-a363e157bcb3-kube-api-access-ftscs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.674657 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.674607 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftscs\" (UniqueName: \"kubernetes.io/projected/fce7e914-5b00-4b9a-905f-a363e157bcb3-kube-api-access-ftscs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.674903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.674666 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-metrics-certs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.674903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.674790 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-webhook-cert\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.677286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.677252 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-webhook-cert\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.677286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.677276 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce7e914-5b00-4b9a-905f-a363e157bcb3-metrics-certs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.736112 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.736084 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftscs\" (UniqueName: \"kubernetes.io/projected/fce7e914-5b00-4b9a-905f-a363e157bcb3-kube-api-access-ftscs\") pod \"maas-controller-84c66fdd7b-kphzc\" (UID: \"fce7e914-5b00-4b9a-905f-a363e157bcb3\") " pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:18.779907 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.779871 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" event={"ID":"06b4a73f-b578-44ea-b4f5-a55650ea6011","Type":"ContainerStarted","Data":"41274a1d36e2d105da04fe829c15bafd880c05b22852e4e29dc927c38360ae71"} Oct 30 20:32:18.780083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.779924 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" event={"ID":"06b4a73f-b578-44ea-b4f5-a55650ea6011","Type":"ContainerStarted","Data":"e95361b5f112de7a99a4de79874c48e73be75c14ebf394de10c264658ae4ecf8"} Oct 30 20:32:18.902447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:18.902399 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podStartSLOduration=2.902384283 podStartE2EDuration="2.902384283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:18.900825484 +0000 UTC m=+3033.744497113" watchObservedRunningTime="2026-10-30 20:32:18.902384283 +0000 UTC m=+3033.746055919" Oct 30 20:32:19.027051 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.027013 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:19.321824 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.321751 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:19.323505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.323473 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:19.323642 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.323520 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:19.418118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.418088 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c66fdd7b-kphzc"] Oct 30 20:32:19.473749 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:32:19.473721 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce7e914_5b00_4b9a_905f_a363e157bcb3.slice/crio-de667fe060e2878e9348f0ac788e938fad3ccd1eaa20053f1f3c0ba669f7fcf3 WatchSource:0}: Error finding container de667fe060e2878e9348f0ac788e938fad3ccd1eaa20053f1f3c0ba669f7fcf3: Status 404 returned error can't find the container with id de667fe060e2878e9348f0ac788e938fad3ccd1eaa20053f1f3c0ba669f7fcf3 Oct 30 20:32:19.785318 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:19.785280 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" event={"ID":"fce7e914-5b00-4b9a-905f-a363e157bcb3","Type":"ContainerStarted","Data":"de667fe060e2878e9348f0ac788e938fad3ccd1eaa20053f1f3c0ba669f7fcf3"} Oct 30 20:32:20.321952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:20.321911 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:20.322351 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:20.321971 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:20.791885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:20.791840 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" event={"ID":"fce7e914-5b00-4b9a-905f-a363e157bcb3","Type":"ContainerStarted","Data":"3915fbcea3b1967aac80ebf3c26c670f1719fe917459958b2c4b67f381ab3e10"} Oct 30 20:32:20.792077 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:20.791915 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:20.911470 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:20.911412 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" podStartSLOduration=2.05198634 podStartE2EDuration="2.911394293s" totalImagesPullingTime="859.407953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:20.910033716 +0000 UTC m=+3035.753705337" watchObservedRunningTime="2026-10-30 20:32:20.911394293 +0000 UTC m=+3035.755065924" Oct 30 20:32:21.321435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:21.321401 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:21.321595 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:21.321456 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:22.321617 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:22.321577 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:22.322124 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:22.321637 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:23.322263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:23.322221 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:23.322650 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:23.322278 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:24.321918 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:24.321879 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:32:24.322108 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:24.321936 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" podUID="06b4a73f-b578-44ea-b4f5-a55650ea6011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:32:26.037275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:26.037239 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:26.037839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:26.037653 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:26.038746 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:26.038722 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-jz4wl" Oct 30 20:32:31.799005 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:31.798977 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c66fdd7b-kphzc" Oct 30 20:32:42.784977 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:32:42.784942 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:32:52.029154 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:32:52.029100 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:12.784404 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:33:12.784343 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:33:17.635121 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.635085 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j"] Oct 30 20:33:17.638491 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.638473 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.684738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.684712 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq"] Oct 30 20:33:17.687919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.687901 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.697126 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697093 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs8x\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-kube-api-access-dbs8x\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697141 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697246 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697224 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697263 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c778d-3136-4cca-a830-6648684db7e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697290 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697313 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697418 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697445 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.697510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.697482 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c778d-3136-4cca-a830-6648684db7e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.744109 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.744081 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j"] Oct 30 20:33:17.752975 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.752947 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq"] Oct 30 20:33:17.797972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.797941 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c870f8-41e5-4488-a43b-744ed41bed3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.797985 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798014 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbq45\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-kube-api-access-cbq45\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798082 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798114 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs8x\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-kube-api-access-dbs8x\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798141 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798159 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798182 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c778d-3136-4cca-a830-6648684db7e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798270 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798354 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798311 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798353 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798449 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798552 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798584 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798632 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798663 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798667 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798715 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.798903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798780 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.798903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798821 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c778d-3136-4cca-a830-6648684db7e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.799117 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.798909 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.799409 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.799389 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c778d-3136-4cca-a830-6648684db7e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.799538 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.799522 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.800754 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.800735 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c778d-3136-4cca-a830-6648684db7e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.800866 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.800845 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c778d-3136-4cca-a830-6648684db7e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.872512 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.872480 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.872611 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.872590 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs8x\" (UniqueName: \"kubernetes.io/projected/d88c778d-3136-4cca-a830-6648684db7e1-kube-api-access-dbs8x\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6rv5j\" (UID: \"d88c778d-3136-4cca-a830-6648684db7e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.899922 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.899902 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.899942 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.899963 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.899993 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c870f8-41e5-4488-a43b-744ed41bed3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900018 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900036 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq45\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-kube-api-access-cbq45\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900058 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900120 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900327 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900156 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900329 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900490 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900392 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900613 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900529 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.900828 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900805 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.901012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.900921 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c870f8-41e5-4488-a43b-744ed41bed3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.902523 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.902499 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.902758 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.902742 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.950953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.950896 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:17.962021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.961994 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:17.962219 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:17.962193 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq45\" (UniqueName: \"kubernetes.io/projected/40c870f8-41e5-4488-a43b-744ed41bed3e-kube-api-access-cbq45\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tkxkq\" (UID: \"40c870f8-41e5-4488-a43b-744ed41bed3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:18.001000 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.000744 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:18.386521 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.386442 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j"] Oct 30 20:33:18.424764 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:33:18.424733 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88c778d_3136_4cca_a830_6648684db7e1.slice/crio-0a765bdabe6a7fd079e9b74aa36a0b119b8c62db0a1efedc077c3501fa9a8120 WatchSource:0}: Error finding container 0a765bdabe6a7fd079e9b74aa36a0b119b8c62db0a1efedc077c3501fa9a8120: Status 404 returned error can't find the container with id 0a765bdabe6a7fd079e9b74aa36a0b119b8c62db0a1efedc077c3501fa9a8120 Oct 30 20:33:18.426889 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.426860 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:18.426972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.426928 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:18.426972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.426960 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:18.446278 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.446259 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq"] Oct 30 20:33:18.449914 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:33:18.449888 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c870f8_41e5_4488_a43b_744ed41bed3e.slice/crio-93db89672c1a630a833e3552b9fab80889c4c5bad550c42ec4738fb1ddd4a206 WatchSource:0}: Error finding container 93db89672c1a630a833e3552b9fab80889c4c5bad550c42ec4738fb1ddd4a206: Status 404 returned error can't find the container with id 93db89672c1a630a833e3552b9fab80889c4c5bad550c42ec4738fb1ddd4a206 Oct 30 20:33:18.452364 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.452331 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:18.452480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.452407 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:18.452480 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:18.452452 4524 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:19.119751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.114976 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" event={"ID":"d88c778d-3136-4cca-a830-6648684db7e1","Type":"ContainerStarted","Data":"5668aec9d1651f914081bda2c5120f12a2587c0f286d35197ee517f6143c0a42"} Oct 30 20:33:19.119751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.115061 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" event={"ID":"d88c778d-3136-4cca-a830-6648684db7e1","Type":"ContainerStarted","Data":"0a765bdabe6a7fd079e9b74aa36a0b119b8c62db0a1efedc077c3501fa9a8120"} Oct 30 20:33:19.122715 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.122668 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" event={"ID":"40c870f8-41e5-4488-a43b-744ed41bed3e","Type":"ContainerStarted","Data":"d7b20c5d4c0ee9a215cbafa5715d4c51f06d117a8c5c7552ebc294fa59741309"} Oct 30 20:33:19.122879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.122754 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" event={"ID":"40c870f8-41e5-4488-a43b-744ed41bed3e","Type":"ContainerStarted","Data":"93db89672c1a630a833e3552b9fab80889c4c5bad550c42ec4738fb1ddd4a206"} Oct 30 20:33:19.238648 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.238307 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podStartSLOduration=2.238293164 podStartE2EDuration="2.238293164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:19.237670509 +0000 UTC m=+3094.081342165" watchObservedRunningTime="2026-10-30 20:33:19.238293164 +0000 UTC m=+3094.081964792" Oct 30 20:33:19.363724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.363525 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podStartSLOduration=2.363503729 podStartE2EDuration="2.363503729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:19.361225645 +0000 UTC m=+3094.204897277" watchObservedRunningTime="2026-10-30 20:33:19.363503729 +0000 UTC m=+3094.207175356" Oct 30 20:33:19.955114 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.955076 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:19.959525 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.959493 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:19.959844 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:19.959814 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:20.003069 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:20.002950 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:20.004770 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:20.004739 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:20.005056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:20.004999 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:20.954722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:20.954099 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:20.954722 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:20.954175 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:21.002268 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:21.002227 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:21.002434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:21.002280 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:21.954218 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:21.954180 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:21.954541 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:21.954495 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:22.001855 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:22.001817 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:22.002423 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:22.002389 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:22.953176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:22.953131 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:22.953476 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:22.953442 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:23.006383 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:23.006347 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:23.007027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:23.006978 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:23.954173 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:23.954135 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:23.954495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:23.954449 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:24.002581 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:24.002532 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:24.002763 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:24.002601 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:24.952093 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:24.952054 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:24.952624 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:24.952120 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:25.001628 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:25.001586 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:25.001825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:25.001652 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:25.956724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:25.953053 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 20:33:25.956724 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:25.953113 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 20:33:26.005554 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:26.005519 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:33:26.005864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:26.005833 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:33:27.955558 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:27.955518 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:27.956013 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:27.955573 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:33:28.003994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:28.003958 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:28.004148 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:28.004015 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:33:28.996927 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:28.996886 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:28.997536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:28.997486 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:33:29.045870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:29.045828 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:29.046006 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:29.045879 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:33:29.889287 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:29.889247 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:29.889884 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:29.889812 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:29.890956 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:29.890930 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" Oct 30 20:33:30.038840 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:30.038803 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:30.039228 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:30.038848 4524 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:33:30.280072 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:30.280040 4524 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:30.280466 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:30.280441 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:30.281183 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:30.281165 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" Oct 30 20:33:42.784992 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:33:42.784953 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:33:52.317034 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:52.316999 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:52.551071 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:52.551040 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:54.920091 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:54.920059 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:59.092849 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:59.092795 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-crkfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:33:59.093496 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:33:59.093446 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-crkfz" podUID="05436fec-ac75-45e2-83aa-90eda66455dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:34:12.923188 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:34:12.923146 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:34:17.913379 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:34:17.913347 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:42.785244 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:34:42.785207 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:34:52.953605 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:34:52.953572 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:12.784769 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:35:12.784729 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:35:24.407630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:35:24.407591 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:31.420972 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:35:31.420933 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:31.754870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:35:31.754762 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:42.784766 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:35:42.784723 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:35:53.096038 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:35:53.096004 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:12.784416 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:36:12.784377 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:36:42.784157 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:36:42.784072 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:36:56.502161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:36:56.502076 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:12.786686 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:37:12.786646 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:37:42.785853 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:37:42.785817 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:37:53.786485 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:37:53.786454 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.784676 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:38:12.784637 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:38:30.755118 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:30.755051 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:31.525084 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:31.525018 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:31.790586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:31.790499 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:36.648565 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:36.648530 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:37.089197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:37.089110 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:37.348099 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:37.348006 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:40.122913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.122872 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:40.390695 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.390667 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:40.593267 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.593230 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:38:40.597364 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.597337 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.657895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.657834 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.657895 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.657883 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg27\" (UniqueName: \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.662434 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.662413 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b2cfa0\"" Oct 30 20:38:40.702862 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.702837 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8144f265-dockercfg-ph9lt\"" Oct 30 20:38:40.704574 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.704554 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:38:40.758519 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.758495 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.758644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.758541 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg27\" (UniqueName: \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.759184 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.759166 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.820925 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.820899 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg27\" (UniqueName: \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\") pod \"payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:40.925894 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:40.925818 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:41.309509 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:41.309483 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:38:41.311428 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:38:41.311398 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ff2d6_3915_4909_9477_c02f91ae0981.slice/crio-8b98586d5ee6664b3873b59d79d47a6fee767e54bdddd60842fbaf8bd46e49b4 WatchSource:0}: Error finding container 8b98586d5ee6664b3873b59d79d47a6fee767e54bdddd60842fbaf8bd46e49b4: Status 404 returned error can't find the container with id 8b98586d5ee6664b3873b59d79d47a6fee767e54bdddd60842fbaf8bd46e49b4 Oct 30 20:38:41.886532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:41.886498 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" event={"ID":"9c8ff2d6-3915-4909-9477-c02f91ae0981","Type":"ContainerStarted","Data":"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a"} Oct 30 20:38:41.886731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:41.886547 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" event={"ID":"9c8ff2d6-3915-4909-9477-c02f91ae0981","Type":"ContainerStarted","Data":"8b98586d5ee6664b3873b59d79d47a6fee767e54bdddd60842fbaf8bd46e49b4"} Oct 30 20:38:41.886731 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:41.886623 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:42.007082 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:42.007032 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" podStartSLOduration=2.007018189 podStartE2EDuration="2.007018189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:42.005805413 +0000 UTC m=+3416.849477033" watchObservedRunningTime="2026-10-30 20:38:42.007018189 +0000 UTC m=+3416.850689817" Oct 30 20:38:42.785615 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:38:42.785578 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:38:42.879079 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:42.879037 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:43.412870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:43.412837 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:48.093330 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:48.093278 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:52.894106 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:52.894073 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:38:53.081199 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:53.081170 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:54.066018 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:54.065987 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:58.163010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:38:58.162972 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.775961 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:01.775925 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.412941 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:02.412910 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.740869 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:02.740790 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.409994 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:03.409958 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:06.256970 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:06.256913 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:08.391986 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:08.391937 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:12.784646 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:39:12.784611 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:39:13.425202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:13.425168 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:18.330243 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:18.330170 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:23.082298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:23.082252 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:27.994411 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:27.994300 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:33.039808 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:33.039769 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:37.978877 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:37.978838 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:42.798161 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:39:42.798036 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:39:42.802192 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:42.802169 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:46.202244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:46.202213 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:47.690193 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:47.690161 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:48.856097 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:48.856063 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:52.964631 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:52.964595 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:53.805244 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:53.805214 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:54.417127 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:54.417092 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:57.998435 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:39:57.998399 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:00.749127 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:00.749087 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:40:01.221903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:01.221869 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:01.452825 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:01.452792 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:02.369391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:02.369354 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:05.750177 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:05.750127 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" containerID="cri-o://fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a" gracePeriod=25 Oct 30 20:40:09.767430 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:09.767381 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:10.063357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:10.063260 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:10.939087 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:10.939050 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:12.785971 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:40:12.785928 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:40:12.892681 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:12.892646 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:40:17.991802 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:17.991764 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:18.230187 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:18.230151 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:18.490382 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:18.490342 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:22.893061 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:22.893014 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:40:25.998306 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:25.998278 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:40:26.058502 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.058441 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\") pod \"9c8ff2d6-3915-4909-9477-c02f91ae0981\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " Oct 30 20:40:26.058625 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.058575 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\" (UniqueName: \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\") pod \"9c8ff2d6-3915-4909-9477-c02f91ae0981\" (UID: \"9c8ff2d6-3915-4909-9477-c02f91ae0981\") " Oct 30 20:40:26.058812 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.058790 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume" pod "9c8ff2d6-3915-4909-9477-c02f91ae0981" (UID: "9c8ff2d6-3915-4909-9477-c02f91ae0981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:26.058913 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.058893 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8ff2d6-3915-4909-9477-c02f91ae0981-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:26.060636 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.060612 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27" pod "9c8ff2d6-3915-4909-9477-c02f91ae0981" (UID: "9c8ff2d6-3915-4909-9477-c02f91ae0981"). InnerVolumeSpecName "kube-api-access-cpg27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:26.160047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.160018 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg27\" (UniqueName: \"kubernetes.io/projected/9c8ff2d6-3915-4909-9477-c02f91ae0981-kube-api-access-cpg27\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:26.368644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.368563 4524 generic.go:356] "Generic (PLEG): container finished" podID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerID="fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a" exitCode=0 Oct 30 20:40:26.368644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.368596 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" Oct 30 20:40:26.368644 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.368613 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" event={"ID":"9c8ff2d6-3915-4909-9477-c02f91ae0981","Type":"ContainerDied","Data":"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a"} Oct 30 20:40:26.368952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.368660 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg" event={"ID":"9c8ff2d6-3915-4909-9477-c02f91ae0981","Type":"ContainerDied","Data":"8b98586d5ee6664b3873b59d79d47a6fee767e54bdddd60842fbaf8bd46e49b4"} Oct 30 20:40:26.368952 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.368686 4524 scope.go:122] "RemoveContainer" containerID="fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a" Oct 30 20:40:26.377745 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.377726 4524 scope.go:122] "RemoveContainer" containerID="fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a" Oct 30 20:40:26.377995 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:40:26.377976 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a\": container with ID starting with fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a not found: ID does not exist" containerID="fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a" Oct 30 20:40:26.378047 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.378004 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a"} err="failed to get container status \"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a\": rpc error: code = NotFound desc = could not find container \"fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a\": container with ID starting with fc7cff61a5f00ab2adb0d5fc25390bcd07067c6ce5298d6c8cdf87732ae0ca0a not found: ID does not exist" Oct 30 20:40:26.533864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.533835 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:40:26.538959 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:26.538936 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b2cfa0-698dd6d96hsvpg"] Oct 30 20:40:27.298052 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:27.298019 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" path="/var/lib/kubelet/pods/9c8ff2d6-3915-4909-9477-c02f91ae0981/volumes" Oct 30 20:40:30.650262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:30.650235 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:31.878920 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:31.878882 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:38.544021 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:38.543987 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:39.266973 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.266940 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 20:40:39.267186 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.267165 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" podUID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" containerName="manager" containerID="cri-o://ad00ae6b99675f6442d133da195e58eb128d15005281723ac7e633226db2ceae" gracePeriod=10 Oct 30 20:40:39.436766 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.436737 4524 generic.go:356] "Generic (PLEG): container finished" podID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" containerID="ad00ae6b99675f6442d133da195e58eb128d15005281723ac7e633226db2ceae" exitCode=0 Oct 30 20:40:39.436958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.436775 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" event={"ID":"d86f8f1d-e792-4041-b3f9-69cdeb26932c","Type":"ContainerDied","Data":"ad00ae6b99675f6442d133da195e58eb128d15005281723ac7e633226db2ceae"} Oct 30 20:40:39.530727 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.530662 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 20:40:39.679742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.679688 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\" (UniqueName: \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\") pod \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " Oct 30 20:40:39.680220 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.679773 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\") pod \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\" (UID: \"d86f8f1d-e792-4041-b3f9-69cdeb26932c\") " Oct 30 20:40:39.680283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.680210 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume" pod "d86f8f1d-e792-4041-b3f9-69cdeb26932c" (UID: "d86f8f1d-e792-4041-b3f9-69cdeb26932c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:39.681878 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.681854 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv" pod "d86f8f1d-e792-4041-b3f9-69cdeb26932c" (UID: "d86f8f1d-e792-4041-b3f9-69cdeb26932c"). InnerVolumeSpecName "kube-api-access-j2jsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:39.781273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.781197 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jsv\" (UniqueName: \"kubernetes.io/projected/d86f8f1d-e792-4041-b3f9-69cdeb26932c-kube-api-access-j2jsv\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:39.781273 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:39.781227 4524 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86f8f1d-e792-4041-b3f9-69cdeb26932c-extensions-socket-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:40.443050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:40.443013 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" event={"ID":"d86f8f1d-e792-4041-b3f9-69cdeb26932c","Type":"ContainerDied","Data":"51b4090a78d828e0183bd686edcb58c5e950b8ecbad4884fac43a28018827c6c"} Oct 30 20:40:40.443050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:40.443030 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln" Oct 30 20:40:40.443328 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:40.443069 4524 scope.go:122] "RemoveContainer" containerID="ad00ae6b99675f6442d133da195e58eb128d15005281723ac7e633226db2ceae" Oct 30 20:40:40.658367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:40.658338 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 20:40:40.708682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:40.708617 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5mqln"] Oct 30 20:40:41.299235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:41.299201 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" path="/var/lib/kubelet/pods/d86f8f1d-e792-4041-b3f9-69cdeb26932c/volumes" Oct 30 20:40:42.784890 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:40:42.784857 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:40:54.806580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:40:54.806544 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:12.784204 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:41:12.784164 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:41:42.784729 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:41:42.784675 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:41:42.957196 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957163 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts"] Oct 30 20:41:42.957566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957552 4524 cpu_manager.go:414] "containerMap: removing container" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" Oct 30 20:41:42.957616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957567 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" Oct 30 20:41:42.957616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957578 4524 cpu_manager.go:414] "containerMap: removing container" podUID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" containerName="manager" Oct 30 20:41:42.957616 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957582 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" containerName="manager" Oct 30 20:41:42.957742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957657 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8ff2d6-3915-4909-9477-c02f91ae0981" containerName="payload-processing" Oct 30 20:41:42.957742 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.957664 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86f8f1d-e792-4041-b3f9-69cdeb26932c" containerName="manager" Oct 30 20:41:42.960759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:42.960743 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.013691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.013648 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-78rg8\"" Oct 30 20:41:43.065363 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.065305 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts"] Oct 30 20:41:43.132298 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.132270 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/9374dcb0-2312-4ee2-9754-242f1a69d324-kube-api-access-rc94j\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.132440 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.132410 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9374dcb0-2312-4ee2-9754-242f1a69d324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.233014 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.232987 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9374dcb0-2312-4ee2-9754-242f1a69d324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.233146 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.233043 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/9374dcb0-2312-4ee2-9754-242f1a69d324-kube-api-access-rc94j\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.233427 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.233406 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9374dcb0-2312-4ee2-9754-242f1a69d324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.294263 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.294235 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc94j\" (UniqueName: \"kubernetes.io/projected/9374dcb0-2312-4ee2-9754-242f1a69d324-kube-api-access-rc94j\") pod \"kuadrant-operator-controller-manager-57bb9bfff-xxcts\" (UID: \"9374dcb0-2312-4ee2-9754-242f1a69d324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:43.574095 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:43.574063 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:44.045806 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:44.045774 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts"] Oct 30 20:41:44.084570 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:41:44.084544 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9374dcb0_2312_4ee2_9754_242f1a69d324.slice/crio-7076a64be147e680cb5233bbb84ab60ccded831e7a065a8175272a4d19be5eea WatchSource:0}: Error finding container 7076a64be147e680cb5233bbb84ab60ccded831e7a065a8175272a4d19be5eea: Status 404 returned error can't find the container with id 7076a64be147e680cb5233bbb84ab60ccded831e7a065a8175272a4d19be5eea Oct 30 20:41:44.719734 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:44.719680 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" event={"ID":"9374dcb0-2312-4ee2-9754-242f1a69d324","Type":"ContainerStarted","Data":"875907f2f92d7c2b2213b7f9832f417a0754617bbdb3f8f22a87bacceaa50e7e"} Oct 30 20:41:44.719950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:44.719853 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:41:44.719950 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:44.719872 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" event={"ID":"9374dcb0-2312-4ee2-9754-242f1a69d324","Type":"ContainerStarted","Data":"7076a64be147e680cb5233bbb84ab60ccded831e7a065a8175272a4d19be5eea"} Oct 30 20:41:44.845738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:44.845661 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" podStartSLOduration=2.845648145 podStartE2EDuration="2.845648145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:44.84422217 +0000 UTC m=+3599.687893801" watchObservedRunningTime="2026-10-30 20:41:44.845648145 +0000 UTC m=+3599.689319773" Oct 30 20:41:55.118323 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:55.118281 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:55.726391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:41:55.726362 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-xxcts" Oct 30 20:42:12.785497 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:42:12.785463 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:42:33.608780 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:33.608678 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:42.362958 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:42.362912 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:42.785512 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:42:42.785474 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:42:43.012810 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:43.011083 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:43.241100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:43.241060 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:46.869837 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:46.869799 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:47.113691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:47.113658 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:50.262968 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:50.262933 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:50.508262 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:50.508230 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:50.928951 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:50.928918 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:42:50.932166 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:50.932149 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:50.984865 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:50.984843 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c530ce\"" Oct 30 20:42:51.013533 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.013502 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jsj\" (UniqueName: \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.013673 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.013553 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.034667 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.034639 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-304f993e-dockercfg-bpnj5\"" Oct 30 20:42:51.036564 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.036539 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:42:51.114058 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.114030 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jsj\" (UniqueName: \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.114180 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.114076 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.114769 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.114750 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.175848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.175826 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jsj\" (UniqueName: \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\") pod \"payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.245433 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.245374 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:51.618102 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:51.618072 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:42:51.620687 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:42:51.620660 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ac3371_d127_4b85_b808_d87468b8195a.slice/crio-5f69d9d4a67c6d04b77edf80df76321cc4312e8758381ece10b698f03b312ad8 WatchSource:0}: Error finding container 5f69d9d4a67c6d04b77edf80df76321cc4312e8758381ece10b698f03b312ad8: Status 404 returned error can't find the container with id 5f69d9d4a67c6d04b77edf80df76321cc4312e8758381ece10b698f03b312ad8 Oct 30 20:42:52.064141 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:52.064104 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" event={"ID":"31ac3371-d127-4b85-b808-d87468b8195a","Type":"ContainerStarted","Data":"c7ffd664accf346dda5b65b776e9a4f7d6c8cf4f6ac49e17f46ccc5880089ac0"} Oct 30 20:42:52.064348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:52.064166 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" event={"ID":"31ac3371-d127-4b85-b808-d87468b8195a","Type":"ContainerStarted","Data":"5f69d9d4a67c6d04b77edf80df76321cc4312e8758381ece10b698f03b312ad8"} Oct 30 20:42:52.064348 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:52.064191 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:42:52.181050 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:52.181004 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" podStartSLOduration=2.180990295 podStartE2EDuration="2.180990295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:52.179926111 +0000 UTC m=+3667.023597741" watchObservedRunningTime="2026-10-30 20:42:52.180990295 +0000 UTC m=+3667.024661991" Oct 30 20:42:53.134730 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:53.132309 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:53.373793 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:53.373762 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:53.612396 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:53.612365 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:55.514479 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:55.514448 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:58.170406 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:42:58.170369 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:03.071919 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:03.071889 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:43:03.124682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:03.124652 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:07.872914 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:07.872870 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:11.884716 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:11.884668 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:12.658864 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:12.658828 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:12.794615 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:43:12.794579 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:43:12.879016 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:12.878933 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:13.096935 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:13.096843 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:16.157967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:16.157923 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:17.662846 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:17.662807 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:22.492929 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:22.492899 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:27.389152 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:27.389120 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:32.171498 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:32.171460 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:32.870829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:32.870796 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:36.987236 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:36.987198 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:41.843059 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:41.843011 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:42.785057 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:43:42.785018 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:43:46.626868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:46.626834 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:51.451566 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:51.451517 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:55.816942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:55.816900 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:56.226916 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:56.226880 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:56.702212 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:56.702180 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:56.918809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:56.918770 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:57.391930 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:43:57.391898 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:01.475634 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:01.475588 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:02.483510 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:02.483478 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:06.440294 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:06.440264 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:09.437516 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:09.437480 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:44:09.756269 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:09.756183 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:09.971319 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:09.971287 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.669853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:10.669818 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:12.785477 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:44:12.785443 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:44:14.390424 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:14.390389 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:14.438679 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:14.438637 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" containerID="cri-o://c7ffd664accf346dda5b65b776e9a4f7d6c8cf4f6ac49e17f46ccc5880089ac0" gracePeriod=25 Oct 30 20:44:14.691870 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:14.691783 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:15.459682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:15.459645 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:22.220359 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:22.220300 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:22.454300 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:22.454265 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:22.688178 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:22.688143 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:23.069796 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:23.069721 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:30.419231 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:30.419191 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.070252 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:33.070218 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:34.574081 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.574049 4524 generic.go:356] "Generic (PLEG): container finished" podID="31ac3371-d127-4b85-b808-d87468b8195a" containerID="c7ffd664accf346dda5b65b776e9a4f7d6c8cf4f6ac49e17f46ccc5880089ac0" exitCode=0 Oct 30 20:44:34.574472 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.574130 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" event={"ID":"31ac3371-d127-4b85-b808-d87468b8195a","Type":"ContainerDied","Data":"c7ffd664accf346dda5b65b776e9a4f7d6c8cf4f6ac49e17f46ccc5880089ac0"} Oct 30 20:44:34.695283 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.695230 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:44:34.791162 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.791134 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\" (UniqueName: \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\") pod \"31ac3371-d127-4b85-b808-d87468b8195a\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " Oct 30 20:44:34.791310 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.791195 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\") pod \"31ac3371-d127-4b85-b808-d87468b8195a\" (UID: \"31ac3371-d127-4b85-b808-d87468b8195a\") " Oct 30 20:44:34.791505 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.791484 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume" pod "31ac3371-d127-4b85-b808-d87468b8195a" (UID: "31ac3371-d127-4b85-b808-d87468b8195a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:34.793202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.793185 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj" pod "31ac3371-d127-4b85-b808-d87468b8195a" (UID: "31ac3371-d127-4b85-b808-d87468b8195a"). InnerVolumeSpecName "kube-api-access-l7jsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:34.892291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.892250 4524 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ac3371-d127-4b85-b808-d87468b8195a-plugins-config-volume\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:44:34.892291 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:34.892285 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jsj\" (UniqueName: \"kubernetes.io/projected/31ac3371-d127-4b85-b808-d87468b8195a-kube-api-access-l7jsj\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:44:35.582809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:35.582688 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" Oct 30 20:44:35.583591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:35.582766 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2" event={"ID":"31ac3371-d127-4b85-b808-d87468b8195a","Type":"ContainerDied","Data":"5f69d9d4a67c6d04b77edf80df76321cc4312e8758381ece10b698f03b312ad8"} Oct 30 20:44:35.583591 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:35.583458 4524 scope.go:122] "RemoveContainer" containerID="c7ffd664accf346dda5b65b776e9a4f7d6c8cf4f6ac49e17f46ccc5880089ac0" Oct 30 20:44:35.750339 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:35.750304 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:44:35.799953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:35.799914 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c530ce-659b9sm8b2"] Oct 30 20:44:37.312682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:37.312642 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ac3371-d127-4b85-b808-d87468b8195a" path="/var/lib/kubelet/pods/31ac3371-d127-4b85-b808-d87468b8195a/volumes" Oct 30 20:44:39.638471 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:39.638438 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:40.626139 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:40.626093 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:42.786293 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:44:42.786258 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:44:56.020027 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:56.019992 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.239100 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:44:56.239072 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:05.990529 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:05.990497 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:10.486592 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:10.486554 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:12.785683 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:45:12.785647 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:45:16.337934 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:16.337882 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.784684 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:45:42.784603 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:45:45.941721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:45.939794 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:45:45.941721 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:45.939867 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:45:46.329883 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:46.329787 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:45:46.330194 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:46.330160 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:45:54.787853 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:54.787812 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:56.517622 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:45:56.517586 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:03.960668 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:03.960631 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:12.785642 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:46:12.785605 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:46:16.336833 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:16.336792 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:46:16.337572 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:16.337538 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:30.936901 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:30.936844 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:46:30.937569 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:30.937516 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tkxkq" podUID="40c870f8-41e5-4488-a43b-744ed41bed3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:31.323917 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:31.323825 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:46:31.324185 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:31.324146 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:42.786244 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:46:42.786200 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:46:56.862609 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:46:56.862525 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:00.516967 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:00.516927 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:00.783903 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:00.783827 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:04.438532 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.438498 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-nx2q2"] Oct 30 20:47:04.439075 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.439054 4524 cpu_manager.go:414] "containerMap: removing container" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" Oct 30 20:47:04.439164 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.439077 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" Oct 30 20:47:04.439235 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.439221 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ac3371-d127-4b85-b808-d87468b8195a" containerName="payload-pre-processing" Oct 30 20:47:04.442151 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.442129 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.508999 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.508967 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-metrics-certs\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.509160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.509013 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-maas-api-tls\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.509160 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.509138 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqsg\" (UniqueName: \"kubernetes.io/projected/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-kube-api-access-jlqsg\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.509437 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.509215 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-openshift-service-ca\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.530890 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.530858 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-nx2q2"] Oct 30 20:47:04.610133 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.610096 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-metrics-certs\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.610299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.610146 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-maas-api-tls\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.610299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.610190 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqsg\" (UniqueName: \"kubernetes.io/projected/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-kube-api-access-jlqsg\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.610299 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.610227 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-openshift-service-ca\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.610910 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.610891 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-openshift-service-ca\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.612630 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.612608 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-maas-api-tls\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.612738 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.612669 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-metrics-certs\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.671272 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.671243 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqsg\" (UniqueName: \"kubernetes.io/projected/44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8-kube-api-access-jlqsg\") pod \"maas-api-595cc648c-nx2q2\" (UID: \"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:04.754380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:04.754311 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:05.133682 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:05.133658 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-nx2q2"] Oct 30 20:47:05.136074 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:47:05.136041 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b2cdbd_d41f_48f4_8d9f_6d6fb44c1dc8.slice/crio-f6860935bb827636942f7c57631ffb20243aa9a7a2b4a1711d3b12a1b62fe9f1 WatchSource:0}: Error finding container f6860935bb827636942f7c57631ffb20243aa9a7a2b4a1711d3b12a1b62fe9f1: Status 404 returned error can't find the container with id f6860935bb827636942f7c57631ffb20243aa9a7a2b4a1711d3b12a1b62fe9f1 Oct 30 20:47:05.137325 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:05.137304 4524 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:47:05.411317 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:05.411278 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" event={"ID":"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8","Type":"ContainerStarted","Data":"f6860935bb827636942f7c57631ffb20243aa9a7a2b4a1711d3b12a1b62fe9f1"} Oct 30 20:47:05.823493 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:05.823400 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:06.046755 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:06.046693 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:06.417036 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:06.417001 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" event={"ID":"44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8","Type":"ContainerStarted","Data":"05dce02c9b2c9eb8251567db6011ff654c5bfe75b16f4f273a6be2c4c6c4c996"} Oct 30 20:47:06.417230 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:06.417079 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:06.536787 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:06.536733 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" podStartSLOduration=1.6735172299999999 podStartE2EDuration="2.536687359s" totalImagesPullingTime="863.170129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:06.534740952 +0000 UTC m=+3921.378412582" watchObservedRunningTime="2026-10-30 20:47:06.536687359 +0000 UTC m=+3921.380358988" Oct 30 20:47:12.432380 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:12.432349 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-nx2q2" Oct 30 20:47:12.602725 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:12.602679 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 20:47:12.602942 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:12.602919 4524 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" podUID="aba9b876-d448-409f-b43e-c4a304e4e4fd" containerName="maas-api" containerID="cri-o://a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8" gracePeriod=30 Oct 30 20:47:12.784545 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:47:12.784465 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:47:13.869882 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.869856 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 20:47:13.953495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.953424 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\") pod \"aba9b876-d448-409f-b43e-c4a304e4e4fd\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " Oct 30 20:47:13.953495 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.953479 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\") pod \"aba9b876-d448-409f-b43e-c4a304e4e4fd\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " Oct 30 20:47:13.953736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.953566 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\" (UniqueName: \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\") pod \"aba9b876-d448-409f-b43e-c4a304e4e4fd\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " Oct 30 20:47:13.953736 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.953634 4524 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\") pod \"aba9b876-d448-409f-b43e-c4a304e4e4fd\" (UID: \"aba9b876-d448-409f-b43e-c4a304e4e4fd\") " Oct 30 20:47:13.953885 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.953857 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca" pod "aba9b876-d448-409f-b43e-c4a304e4e4fd" (UID: "aba9b876-d448-409f-b43e-c4a304e4e4fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:13.954051 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.954033 4524 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba9b876-d448-409f-b43e-c4a304e4e4fd-openshift-service-ca\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:13.956168 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.956139 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls" pod "aba9b876-d448-409f-b43e-c4a304e4e4fd" (UID: "aba9b876-d448-409f-b43e-c4a304e4e4fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:13.956286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.956165 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs" pod "aba9b876-d448-409f-b43e-c4a304e4e4fd" (UID: "aba9b876-d448-409f-b43e-c4a304e4e4fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:13.956286 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:13.956260 4524 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9" pod "aba9b876-d448-409f-b43e-c4a304e4e4fd" (UID: "aba9b876-d448-409f-b43e-c4a304e4e4fd"). InnerVolumeSpecName "kube-api-access-8bwh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:14.055140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.055107 4524 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bwh9\" (UniqueName: \"kubernetes.io/projected/aba9b876-d448-409f-b43e-c4a304e4e4fd-kube-api-access-8bwh9\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:14.055140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.055136 4524 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-metrics-certs\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:14.055140 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.055147 4524 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aba9b876-d448-409f-b43e-c4a304e4e4fd-maas-api-tls\") on node \"ip-10-0-130-41.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:14.509551 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.509517 4524 generic.go:356] "Generic (PLEG): container finished" podID="aba9b876-d448-409f-b43e-c4a304e4e4fd" containerID="a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8" exitCode=0 Oct 30 20:47:14.509757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.509567 4524 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" Oct 30 20:47:14.509757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.509606 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" event={"ID":"aba9b876-d448-409f-b43e-c4a304e4e4fd","Type":"ContainerDied","Data":"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8"} Oct 30 20:47:14.509757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.509660 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm" event={"ID":"aba9b876-d448-409f-b43e-c4a304e4e4fd","Type":"ContainerDied","Data":"be5a4a49e48c3ffabeb868a08a43e9e43ec38dc4c721001a87b5b0a071948c46"} Oct 30 20:47:14.509757 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.509688 4524 scope.go:122] "RemoveContainer" containerID="a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8" Oct 30 20:47:14.520372 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.520353 4524 scope.go:122] "RemoveContainer" containerID="a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8" Oct 30 20:47:14.520668 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:47:14.520647 4524 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8\": container with ID starting with a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8 not found: ID does not exist" containerID="a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8" Oct 30 20:47:14.520809 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.520677 4524 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8"} err="failed to get container status \"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8\": rpc error: code = NotFound desc = could not find container \"a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8\": container with ID starting with a158cb420385a1afd34be5a969d6858fd39bf98f332d228f75a40f08afa6edd8 not found: ID does not exist" Oct 30 20:47:14.676003 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.675974 4524 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 20:47:14.726083 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:14.726057 4524 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vwdcm"] Oct 30 20:47:15.297879 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:15.297849 4524 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba9b876-d448-409f-b43e-c4a304e4e4fd" path="/var/lib/kubelet/pods/aba9b876-d448-409f-b43e-c4a304e4e4fd/volumes" Oct 30 20:47:42.784850 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:47:42.784811 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:47:46.333748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:46.324900 4524 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:46.333748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:46.324973 4524 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6rv5j" podUID="d88c778d-3136-4cca-a830-6648684db7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:57.296553 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:47:57.296518 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:12.924899 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:48:12.924857 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:48:23.585010 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:23.584978 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:48:24.779872 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:24.779839 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-fjxz5_57a3ea9e-fd4d-4623-81f9-91732b0c6cf9/manager/0.log" Oct 30 20:48:25.439691 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:25.439653 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt_bca56e42-576b-4d19-a433-7612128037e8/manager/0.log" Oct 30 20:48:25.507215 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:25.507194 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt_bca56e42-576b-4d19-a433-7612128037e8/copy-manifests/0.log" Oct 30 20:48:27.047264 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:27.047184 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-nx2q2_44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8/maas-api/0.log" Oct 30 20:48:29.281197 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:29.281161 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kk59t_706e28a3-112e-4d31-bafa-57f173d640f7/postgres/0.log" Oct 30 20:48:31.709042 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:31.709011 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-bkrk8_79bcaf70-b3ba-4651-941b-4e2f6e6e93eb/manager/0.log" Oct 30 20:48:32.285748 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:32.285695 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-xxcts_9374dcb0-2312-4ee2-9754-242f1a69d324/manager/0.log" Oct 30 20:48:33.045819 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:33.045789 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rs8pk_db72c297-f924-45ba-aa81-81a0f9945c13/manager/0.log" Oct 30 20:48:34.195008 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:34.194975 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jc4d2_8fa38359-9276-47a8-9f17-fb94689237e1/discovery/0.log" Oct 30 20:48:34.699656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:34.699626 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64985d6c8c-g96kl_7d633935-0edc-4d93-8496-6617f6b98f41/kube-auth-proxy/0.log" Oct 30 20:48:35.485563 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:35.485525 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6rv5j_d88c778d-3136-4cca-a830-6648684db7e1/istio-proxy/0.log" Oct 30 20:48:35.771852 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:35.771777 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-crkfz_05436fec-ac75-45e2-83aa-90eda66455dc/istio-proxy/0.log" Oct 30 20:48:36.584580 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:36.584553 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-jz4wl_06b4a73f-b578-44ea-b4f5-a55650ea6011/istio-proxy/0.log" Oct 30 20:48:37.129341 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:37.129310 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-tkxkq_40c870f8-41e5-4488-a43b-744ed41bed3e/istio-proxy/0.log" Oct 30 20:48:39.346677 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:39.346647 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-x4wvm_36dc7fbd-ec30-4beb-8b63-6c82b7948da4/main/0.log" Oct 30 20:48:39.629759 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:39.629664 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-mnfw4_6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9/main/0.log" Oct 30 20:48:39.864829 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:39.864799 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7d7d5_e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1/main/0.log" Oct 30 20:48:40.394499 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:40.394469 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc_6f2e017c-f9f6-4850-a8c1-63e34c1706f9/main/0.log" Oct 30 20:48:42.784185 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:48:42.784140 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:48:52.671848 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:52.671818 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:48:53.735367 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:53.735340 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-fjxz5_57a3ea9e-fd4d-4623-81f9-91732b0c6cf9/manager/0.log" Oct 30 20:48:54.513157 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:54.513129 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt_bca56e42-576b-4d19-a433-7612128037e8/copy-manifests/0.log" Oct 30 20:48:54.580953 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:54.580920 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-cpgdt_bca56e42-576b-4d19-a433-7612128037e8/manager/0.log" Oct 30 20:48:55.971536 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:55.971505 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-nx2q2_44b2cdbd-d41f-48f4-8d9f-6d6fb44c1dc8/maas-api/0.log" Oct 30 20:48:57.500054 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:57.500004 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:57.918944 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:57.918915 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kk59t_706e28a3-112e-4d31-bafa-57f173d640f7/postgres/0.log" Oct 30 20:48:59.832645 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:48:59.832615 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-bkrk8_79bcaf70-b3ba-4651-941b-4e2f6e6e93eb/manager/0.log" Oct 30 20:49:00.320431 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:00.320399 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-xxcts_9374dcb0-2312-4ee2-9754-242f1a69d324/manager/0.log" Oct 30 20:49:01.070839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:01.070814 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rs8pk_db72c297-f924-45ba-aa81-81a0f9945c13/manager/0.log" Oct 30 20:49:02.005378 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:02.005347 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jc4d2_8fa38359-9276-47a8-9f17-fb94689237e1/discovery/0.log" Oct 30 20:49:02.502275 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:02.502247 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64985d6c8c-g96kl_7d633935-0edc-4d93-8496-6617f6b98f41/kube-auth-proxy/0.log" Oct 30 20:49:03.343165 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:03.343136 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6rv5j_d88c778d-3136-4cca-a830-6648684db7e1/istio-proxy/0.log" Oct 30 20:49:03.628627 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:03.628549 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-crkfz_05436fec-ac75-45e2-83aa-90eda66455dc/istio-proxy/0.log" Oct 30 20:49:04.435056 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:04.435026 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-jz4wl_06b4a73f-b578-44ea-b4f5-a55650ea6011/istio-proxy/0.log" Oct 30 20:49:05.015518 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:05.015484 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-tkxkq_40c870f8-41e5-4488-a43b-744ed41bed3e/istio-proxy/0.log" Oct 30 20:49:07.034176 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:07.034148 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-x4wvm_36dc7fbd-ec30-4beb-8b63-6c82b7948da4/main/0.log" Oct 30 20:49:07.319821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:07.319744 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-mnfw4_6493447a-ee7e-42b6-b1fa-9e5b0fc7d6c9/main/0.log" Oct 30 20:49:07.554821 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:07.554785 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7d7d5_e7487f7e-ed57-4a3a-aa9d-dc78d5f13da1/main/0.log" Oct 30 20:49:08.058579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:08.058503 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-mgxsc_6f2e017c-f9f6-4850-a8c1-63e34c1706f9/main/0.log" Oct 30 20:49:12.784735 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:49:12.784668 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:49:37.593685 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:37.593647 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:49:38.181839 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:38.181801 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:49:42.785139 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:49:42.785103 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:49:57.917447 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:49:57.917368 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.784713 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:50:12.784657 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:50:16.870357 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:16.870313 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:50:17.449965 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:17.449930 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c66fdd7b-kphzc_fce7e914-5b00-4b9a-905f-a363e157bcb3/manager/0.log" Oct 30 20:50:19.001391 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:19.001356 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:19.253750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:19.253644 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:28.501892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:28.501861 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jqblf_cdfc0cb5-a33f-488c-974e-75c43659c08a/global-pull-secret-syncer/0.log" Oct 30 20:50:31.065132 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:31.065092 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" Oct 30 20:50:32.602751 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:32.602721 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-41.us-west-1.compute.internal_baf763e8cc3376af1056e0b1dfc87c5b/haproxy/0.log" Oct 30 20:50:37.763436 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.763399 4524 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj"] Oct 30 20:50:37.763868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.763816 4524 cpu_manager.go:414] "containerMap: removing container" podUID="aba9b876-d448-409f-b43e-c4a304e4e4fd" containerName="maas-api" Oct 30 20:50:37.763868 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.763857 4524 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba9b876-d448-409f-b43e-c4a304e4e4fd" containerName="maas-api" Oct 30 20:50:37.763997 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.763957 4524 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba9b876-d448-409f-b43e-c4a304e4e4fd" containerName="maas-api" Oct 30 20:50:37.767040 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.767021 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:37.824205 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.824182 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj"] Oct 30 20:50:37.871044 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.871024 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-bkrk8_79bcaf70-b3ba-4651-941b-4e2f6e6e93eb/manager/0.log" Oct 30 20:50:37.872094 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.872078 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"kube-root-ca.crt\"" Oct 30 20:50:37.872161 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.872101 4524 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"openshift-service-ca.crt\"" Oct 30 20:50:37.880005 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.879990 4524 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t2ndh\"/\"default-dockercfg-zhp6b\"" Oct 30 20:50:37.933336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.933311 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-sys\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:37.933452 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.933370 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466q4\" (UniqueName: \"kubernetes.io/projected/1623fcac-f84d-4c8c-96c9-e5c1908a639a-kube-api-access-466q4\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:37.933507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.933448 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-proc\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:37.933507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.933479 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-lib-modules\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:37.933507 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:37.933501 4524 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-podres\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034512 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466q4\" (UniqueName: \"kubernetes.io/projected/1623fcac-f84d-4c8c-96c9-e5c1908a639a-kube-api-access-466q4\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034576 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034552 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-proc\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034579 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-lib-modules\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034596 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-podres\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034641 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-proc\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034668 4524 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-sys\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034718 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-sys\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034722 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-podres\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.034798 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.034666 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1623fcac-f84d-4c8c-96c9-e5c1908a639a-lib-modules\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.095785 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.095767 4524 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466q4\" (UniqueName: \"kubernetes.io/projected/1623fcac-f84d-4c8c-96c9-e5c1908a639a-kube-api-access-466q4\") pod \"perf-node-gather-daemonset-msdjj\" (UID: \"1623fcac-f84d-4c8c-96c9-e5c1908a639a\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.379012 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.378935 4524 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:38.734261 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.734228 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-xxcts_9374dcb0-2312-4ee2-9754-242f1a69d324/manager/0.log" Oct 30 20:50:38.751548 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:38.751522 4524 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj"] Oct 30 20:50:38.753354 ip-10-0-130-41 kubenswrapper[4524]: W1030 20:50:38.753332 4524 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1623fcac_f84d_4c8c_96c9_e5c1908a639a.slice/crio-79a9b6d64902293d0f531d49b657db2f90eae9339f443ec161b008aa182dcb1c WatchSource:0}: Error finding container 79a9b6d64902293d0f531d49b657db2f90eae9339f443ec161b008aa182dcb1c: Status 404 returned error can't find the container with id 79a9b6d64902293d0f531d49b657db2f90eae9339f443ec161b008aa182dcb1c Oct 30 20:50:39.533336 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:39.533300 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" event={"ID":"1623fcac-f84d-4c8c-96c9-e5c1908a639a","Type":"ContainerStarted","Data":"368d5044d9f5f528af943b2308e30926bbf00c539697e59704706f19e409685f"} Oct 30 20:50:39.533750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:39.533351 4524 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" event={"ID":"1623fcac-f84d-4c8c-96c9-e5c1908a639a","Type":"ContainerStarted","Data":"79a9b6d64902293d0f531d49b657db2f90eae9339f443ec161b008aa182dcb1c"} Oct 30 20:50:39.533750 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:39.533419 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:39.649586 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:39.649534 4524 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" podStartSLOduration=2.649520163 podStartE2EDuration="2.649520163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:39.649067601 +0000 UTC m=+4134.492739232" watchObservedRunningTime="2026-10-30 20:50:39.649520163 +0000 UTC m=+4134.493191795" Oct 30 20:50:39.842629 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:39.842559 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rs8pk_db72c297-f924-45ba-aa81-81a0f9945c13/manager/0.log" Oct 30 20:50:42.785179 ip-10-0-130-41 kubenswrapper[4524]: E1030 20:50:42.785142 4524 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w9tj4" containerName="extractor" Oct 30 20:50:45.547202 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:45.547173 4524 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-msdjj" Oct 30 20:50:48.120656 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:48.120619 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dc8mn_f313d87a-d788-4a3e-9e2a-43ea389e9a54/node-exporter/0.log" Oct 30 20:50:48.538658 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:48.538614 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dc8mn_f313d87a-d788-4a3e-9e2a-43ea389e9a54/kube-rbac-proxy/0.log" Oct 30 20:50:48.957899 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:48.957873 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dc8mn_f313d87a-d788-4a3e-9e2a-43ea389e9a54/init-textfile/0.log" Oct 30 20:50:56.194579 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:56.194553 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-s9zjk_28f6fd9e-fe8a-4e23-87d8-38a68e9d1448/prometheus-operator/0.log" Oct 30 20:50:56.609892 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:56.609814 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-s9zjk_28f6fd9e-fe8a-4e23-87d8-38a68e9d1448/kube-rbac-proxy/0.log" Oct 30 20:50:57.033420 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:57.033392 4524 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-l2n47_6cf5d827-96a6-4b4b-a285-7021415f5b65/prometheus-operator-admission-webhook/0.log" Oct 30 20:50:58.367249 ip-10-0-130-41 kubenswrapper[4524]: I1030 20:50:58.367212 4524 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h2ktl_e14438f5-ecd7-4a84-9bb4-7648b87bd075/konnectivity-agent/0.log" limit=1024